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Abstract 


By  looking  at  the  simple  task  of  tossing  a  l)ean  bag  from  hand  to  hand,  we  show  how  tin'  macro 
operator  method  breaks  down  wlien  formula  ing  agent  models  that  interact  with  an  iinc(Mlain 
e.Kternal  world.  .V  macro  operator  encapsulates  a  plan  to  reach  an  ol) jectivi'.  Occasionally  the 
objective  wiU  be  found  to  be  unachievable,  requiring  the  macro  operator  and  its  plan  to  be  rf'jected. 
Letting  the  macro  operator  interact  with  thee.xternal  world  does  not.  by  itself,  change  this  situation, 
but  the  fact  that  the  results  of  the  interaction  are  uncertain,  and  tlu'  ag('nl‘s  knowl('dg('  incomi)lete. 
does.  The  key  idea  is  that  the  agent  can't  positively  determine  if  [uogress  towards  the  ol)jeclive 
is  being  made  in  the  external  world,  and  thus  errors  will  be  math'  in  r<'j('cting  a  m;>cro  operator 
that  would  succeed.  VVe  show  that  there  are  a  number  of  methods  by  which  i  Ih'  agent  c;ui  recover 
from  such  an  operator  rejection  and  continiH'  toward  the  operator's  ob jectivi'.  If  wi'  make  opertitor 
rejection  and  recovery  into  a  common  m<*chanism.  then  thi'  opi'iators  and  tin'  plans  t  hi'v  repri'sent 
will  be  split  by  the  interaction  into  a.  seipience  of  smaller  o|)erators  I'ach  doing  a  portion  of  the 
work  toward  the  objective  of  the  larger  operator. 

The  models  are  described  in  terms  of  Soar  and  we  assume  thi'  reader’s  familiarity  with  both  t  hi- 
architecture  [Laird  and  Rosenbloom.  l!)H7l  and  the  |*rol)h'm  S|)ace  Computational  .Model  [.Newi'll 
c/  «/..  1991]  in  our  discussions. 
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1  Introduction 


In  this  paper  we  examine  an  agent  interacting  with  an  uncertain  external  world  and  how  tliis 
interaction  constrains  a  model  of  the  agent.  In  particular,  we  look  at  agent  models  formulated 
as  problem  spaces  in  the  Problem  Space  ( 'omputational  Model  (I’SCM)  [.N’ewell  <  /  al..  l()!)l]  and 
implemented  in  Soar  [Laird  and  Ro.senbloom.  1!)H7].  and  we  assume  the  reader  is  familiar  with 
these  systems.  In  both  the  PSCM  and  Soar  an  objective  (or  goal)  is  a  definition  of  some  state  of 
both  the  model  and  the  external  world,  and  an  ojrerator  is  what  causes  movement  towards  some 
objective.  Each  operator  has  its  own  objective,  that  hopefully  is  a  ste])  toward  achieving  .some 
larger  objective.  Given  a  particular  operator,  the  amount  of  interaction  it  can  do  with  the  external 
world  depends  on  both  the  reliability  of  the  external  world  in  producing  a  resjronse  to  a  re(|uest<>d 
action,  and  the  speed  of  the  external  world  in  producing  the  res[)on.se.  I'his  limit  on  tin'  amount 
of  interaction  would  seem  to  dmit  the  size  of  the  o|)err.tor‘s  objective,  what  the  operator  is  Irving 
to  achieve.  However,  we  will  show  that  instead,  an  o[*erator's  ohjc'ctivc'  ran  be  shaix’d  across  manv 
operators  each  making  some  progress.  In  particular,  given  an  objective  that  is  too  large,  we  will 
show  methods  for  automatically  splitting  the  (rrocessing  of  the  objc'ctive  into  a  sequence  of  ^nailer 
operators  so  that  the  objective  is  still  achieved. 

We  start  with  the  fact  that  actions  in  the  external  world  take  tiiiu'.  and  models  that  r('(|n<'si 
actions  of  the  world  have  to  do  something  while  the  external  world  is  producing  a  desiix'd  res|)on.s('. 
(.’urrently.  most  models  wait  for  a  response  or  check  that  i)rogress  towards  a  resi)ons('  is  being 
made.  Since  a  desired  response  is  not  always  |)roduced.  a  method  of  t('rniinat ing  iIk'  waiting  has  to 
be  available.  The  basis  for  this  termination  is  a  form  of  ('xhaiisi ion.  and  the  knowledge  g('nerati'd 
from  it  can  be  overgeneral,  applying  in  other  situations  when  waiting  would  be  more  tipproitriiite. 
However,  we  will  de.scribe  a  range  of  methods  that  can  recover  from  an  imippropriale  termination 
w'ith  another  operator  that  can  reach  the  original  objective'. 

The  combination  of  teriidnation  |)o.ssibly  followed  by  recove'iv  w('  call  spliUnig.  Splitting  is  a 
run-time  strategy  for  a  model,  ami  is  an  alternative'  to  large  ope'iators  ih.it  w.iii  for  .■ichii'ving  .iii 
objective.  Waiting  still  occurs  in  syste'ms  that  split  the  ofierator.  However,  the  w.iiting  no  longer 
occurs  within  the  monolithic  operator,  but  rather  betwee'n  the  smaller  split  o|)('raiors.  This  allows 
work  to  proceed  on  other  tasks  while  waiting,  by  simply  picking  oiiertilors  lor  thosi'  tasks.  This 
type  of  multi-tasking  occurs  without  the'  umb'sirable  task  composition  that  oicurs  when  jiicking  tin' 
operators  for  a  second  task  within  a  large  waiting  operator.  The  downside  of  splitting  is  t  In'  extra 
work  done  for  recovery.  .Although  a  particular  Soar  or  I’SC.M  modi'l  might  mix  split  and  monolithic 
operators  to  achieve  some  performance  goal,  we  will  show  that  thi'  mechanisms  for  handling  splits 
due  to  failure  of  the  external  world  producing  a  desired  res|)oiis('  must  exist.  Sottie  method  of 
recovery  from  overgeneral  ti’rminations  tnitst  exist  as  well. 

The  application  of  ;in  operator  is  considi'ced  couqih'ie  u  heti  the  o|)eraior's  ob  jecii\<'s  h.i\e 
been  achieved.  Throughout  this  pupor.  ;i  fanatiml  /»»*</»/ ol  1*S( '\i  atid  Soar  opi'iator  applicatioti  is 
assumed.  This  means  that  once  a  I’SCM  or  Soar  operator  is  select «'d.  t  he  complet ion  of  t  In'  op<'rator 
will  not  be  inhibited  by  the  architecture  of  either  the  I’SC.M  or  Soar.  I  bis  is  our  ititi'rpretation  of 
the  definition  of  a  PSC.M  oirerator  ap|)licatiou  as  "an  effective  ju-oci'diire  lor  a  littn  tion  '  [.Nh'wc'll  <  / 
III..  IWH]. 

fhe  fanatical  completion  assum[)tion  is  why  soun'  form  of  le.n  ning  I'pemtoi  iei  imti.it  ions  .md 
recovery  from  over-general  operator  terminations  must  exist  in  I’St  M  and  ''.o.n  modi'Is,  W  hen  t  In- 
aihievenn'iit  of  an  objective  is  kiioirn  to  be  impossible.  I  heti  the  opemioi  .it  tempt  mg  to  .nhim..' 
t  hat  objective  must  be  tertninated.  I'his  tertuinal  ion  redelines  the  ob  jei  1 1  w  ol  the  opec.itoi  so  t  h.il 
t  he  opi'rator  is  considered  complete  in  this  new  i.i.sc-.  I  his  tei  tnin.il  ion  nn’.h.imsm  In  itse||  .loes 
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not  violate  the  fanatical  completion  assumption.  However,  if  a  mistake  is  made  in  ap|)lyinfj;  the 
learned  termination  knowledge,  then  it  must  be  |)ossible  to  achieve  the  objective  ('ven  after  the 
operator  is  terminated  or  the  fanatical  completion  tassumption  will  l)e  violated.  I'liis  is  wliy  some 
method  of  recovery  is  required. 

In  the  rest  of  the  paper  we  describe  how  interacting  with  the  external  world  acts  as  a  force 
for  splitting  PS(,'M  and  Soar  operators  into  smaller  operators.  VVe  start  by  looking  at  how  Soar  s 
knowledge  compilation  mechanism  can  cause  operators  to  split  in  Section  2.  Since  knowledge 
compilation  is  within  the  architecture,  these  splits  occur  in  a  model  without  the  Soar  agent  doing 
any  deliberate  processing  to  invoke  a  split.  Why  this  occurs  and  how  to  recover  in  these  situations 
is  the  major  point  of  the  discussion.  In  Section  -i  we  discuss  the  delilxnate  splitting  of  ()|)erators. 
and  the  implications  deliberate  splitting  ha.s  for  recovery.  In  general,  delilx'ratc*  si)litting  nMiioves 
more  information  than  architectural  splitting,  making  some  forms  of  recoveiy  more  didicnlt.  I'he 
last  topic,  described  in  Section  -1.  involves  PSC.M  models  that  are  doing  ninitipk'  tasks  and  include 
a  goal  of  high  utilization  of  the  cognitive  resource.  Making  the  PSCM  model's  use  of  the  cognitive 
resource  more  efficient  means  that,  whenever  possible,  work  on  one  of  possibly  multiple  tasks  is 
being  done.  If  a  task's  progress  depends  on  some  external  world  respotis<'.  then  rather  than  wait 
for  that  response  a  different  task  should  bo  worked  on.  We  discuss  how  splitting  ludps  achieve  this 
goal,  and  also  discuss  other  ways  to  handle  multiple  tasks  and  their  relation  to  splittitig.  finally. 
Section  -o  provides  a  discu.ssion  of  how  these  tojHcs  lit  together  and  into  the  general  PSCM  and 
Soar  picture. 

2  Architectural  splitting  of  operators  in  Soar 

Soar  is  an  implementation  language  for  PSCM  imnlels.  However.  Soar  only  ai)i)roximat(>s  some 
PSCM  functionality,  in  particular  PSCM  learning.  PSC.M  learning  occurs  when  an  im|)asse  occurs 
in  a  problem  space,  the  upper  problem  sfrace.  ami  another  |)roblem  space,  tiu'  /ouu  r  problem  s|)ace 
provides  the  knowlerlge  that  resolves  the  impasse.  Our  simirle  model  of  PSCM  h'arning  is:  Lower 
problem  space  knowledge  is  translated  into  a  useful  immediately  applicable  form  for  the  ui)p('r 
problem  space.  This  translation  is  assumed  to  be  |)erfect.  llowi’Vt'r.  because'  i  hi'  dehnition  ol 
PSCM  knowledge  is  too  vagui’.  PSCM  learning  is  also  not  wi'll  spi'cilied.  Thus  Soar  is  h'l’l  with 
attempting  to  implement  perfect  PSCM  learning  without  any  di'iailed  guid.ince  from  the  PSCM. 
Soar's  learning  mechanism,  clninkiiui.  fails  to  be  ju'ifect  in  certain  cases.  I'his  section  describi's  the 
effects  of  this  learning  failure  on  Soar  inoilels.  and  liow  Soar  models  can  recover  from  such  a  failure. 

Cliunking  fails  when  the  working  im'inory  information  in  the  upper  problem  spaci'  changes  whih' 
the  lower  problem  space  is  resolving  the  impa.s.se.  In  particular,  i  hunking  assumes  that  when  .i  lower 
l)roblem  space  decision  is  based  on  working  metnory  information  in  the  up|>er  problem  space,  either 
that  upper  |)roblem  space  information  stays  constant,  or  the  results  of  the  decision  ari'  not  used  to 
construct  a  chunk.  We  will  call  a  situation  wliere  a  lower  problem  s|)ace  has  made  a  decision  ba.sed 
on  knowledge  in  the  upper  probh'iu  space  and  that  information  has  changi'd  h  inimnilhi  iiiroD.-i.-h  iil. 

I  emporally  inconsistent  sit  uat  ions  can  arise  only  if  t  he  /w  rsish  nci  of  a  decision  exci'eds  t  he  original 
reasons  that  supported  the  decision.  .\  cliiink.  created  frotii  a  tettiporally  iticonsisienl  sit  nation  is 
called,  a  iioii-roiilt  niiMtmtii  oils  iliiuik  because  all  tin'  workitig  itii'tuory  inlortmttiou  in  the  itp|)er 
probletti  space  used  tocotistntct  the  chunk  is  ttol  present  at  I  hi'  time  the  chitnk  was  consiruited. 

.Non-cotitetnporani'oits  chittiks  are  a.  probletti  Itecaiise  they  umy  not  apply  in  the  desired  situa 
tioiis.  tints  causing  an  impasse  to  occur  again.  .\  chunk's  conditions  are  formed  from  the  working 
memory  information  that  led  to  its  creation.  Hy  deiiiiition.  the  temporttlly  iticonsistent  workitig 
memitry  information  is  not  available  in  the  iipfter  problem  problem  space  when  the  chunk  is  created. 


Thus,  the  chunk  cannot  apply  to  the  exact  information  that  created  it.  It  can  apply  to  similar 
information  if  it  is  available.  However,  non-contemporaneoiis  chunks  might  never  a|)i)ly.  and  thus 
no  knowledge  transfer  occurs.  As  an  e.xample.  wo  later  show  a  non-contemporaneous  chunk  as¬ 
sociated  with  catching  a  bean  bag  where  the  bean  bag  has  to  be  in  tlie  left  hand  to  be  caught 
by  the  right  hand.  This  chunk  will  never  apply.  .Since  persistence  is  iec|uired  for  a  temporally 
inconsistent  situation  to  e.xist  and  persistence  is  an  attribute  of  operator  application,  most  non- 
contemporaneous  chunks  occur  in  learning  operator  application  knowledge.  The  failure  to  apply 
during  operator  application  usually  causes  some  chain  of  data  dependent  chunks  to  also  fail.  This 
failure  splits  the  learned  operator  application  knowledge  into  two  com|)onents:  tinjse  that  can  apply 
before  the  non-contemporaneous  chunk  and  tho.se  that  cannot  apply.  J'lie  non-application  of  this 
chain  of  chunks  can  cause  an  impasse  to  occur  again.  I'he  method  of  recovering  from  such  splits 
is  to  restore  the  context  of  the  problem  solving  so  that  the  operators  that  originally  applied  in  the 
impasse  will  apply  again.  The  details  of  this  process  are  the  main  subject  of  this  s('ction. 

One  important  fact  to  note  before  we  continue  is  that  t  he  sit  uat  ions  I  hat  produce  noti-contemporaneoiis 
chunks  are  unavoidable  when  interacting  with  adynamic  ('Xtemal  world.  I’ersistfuicc'  is  recjuired  for 
temporally  inconsistent  situations  to  exist,  and  Soar  operators  show  such  pmsistenci'.  Tin'  pickin» 
of  an  operator  in  the  lower  problem  space  can  depend  upon  ('xtf'rnal  world  objects  accessed  via 
the  upper  problem  space.  Since  the  external  world  can  change  these  upper  probhun  s|)ac(>  objects 
at  any  time,  and  the  operator  jtersists  aiul  can  be  tested  aftru'  t  lu'se  changes  occurred,  we  can't 
eliminate  all  the  situations  that  produce  chunks.  .Also,  since  the  exti'rnal  world  is  changing  the 
reasons  supporting  the  operator's  initial  proposal,  we  can't  in  gmieral  prcslict  when  temporally 
inconsistent  situations  will  occur.  We  can  change  the  Soar  architc'ct  iin'  to  modify  the  (hdinifion  of 
persistence  so  temporal  inconsistencies  do  not  occur.  This  is  called  S-supi)ort  [l.aird  and  Huffman. 

199'2]  and  how  it  affects  the  topic  of  splitting  is  ih'scrilx'd  in  Section  2.1.  llowmer.  we  will  argue 
that  S-support  by  itself  doesn't  change  the  (licfiire.  it  is  just  a  di/f'ermit  mechanism  lor  enforcing  the 
current  paradigm.  It  is  an  eager  method  that  detects  tmnporal  inconsisimicic's  foiling  the  user  to 
consider  them  when  they  occur.  .N’on  conti'inporaueous  chunks  turns  out  to  be  a  la/y  method  that 
forces  the  user  to  handle  tmiiporal  inconsistmicies  wlimi  the  agent  aitmnpts  to  use  the  inapplicable 
learned  knowledge. 


2.1  Cyclic  toss  task 


\W  will  discuss  these  issues  through  the<>.\amph*  task  of  tossing  a  bean  bag  from  one  hand  to 
the  other  and  back,  a  cyclic  toss.  In  solving  this  task,  we  will  use  the  following  tliK'e  operators; 
cyclic-toss.  toss,  and  catch.  .V  short  des<ri[)t ion  of  these  operators  is  shown  in  I  igiir<'  1.  The 
definition  of  the  cyclic-toss  o[)erator  in  l  igiire  I  dix'sii'i  include  tin'  knowh'dgi'  of  iiow  to  get  the 
bean  bag  from  one  hanil  to  tin'  otluT.  it  only  has  the  knowledge  lo  note  t  hat  i  he  inierm('diat('  state 
is  achieved.  When  we  (h'scribe  the  execution  of  the  <  vclic  loss  task.  w<'  will  show  the  cyclic  toss 
operator  learnijig  the  other  knowledge  recpiired  to  adiieve  its  olijeclivi'. 

In  Figure  1,  we  use  the  terms  itncontlilinns  and  nhlirlin  lo  dislingnish  portions  of  tin'  (oiidi- 
tions  of  the  pmdurtir)ns  that  implement  the  operator.  I’rc-condil ions  are  a  common  phinning  i<'rm 
[hikes  and  .'Vilsson.  1971].  and  we  are  using  it  in  exactly  the  sami'  manner  as  the  plaii'iing  lileralnri'. 
I’recomiitions  are  rlilfereni iaied  from  the  [)roposal  («indilions  for  an  opmalor.  because'  an  opi'ralor's 
prf)pf)sal  conditions  slate  that  this  ope-ralor  is  applicable  to  I  h<'  probh'in  at  haml.  and  the  pre'con 
ditions  state*  I  hat  t  his  ope*r;i,lor  can  a|)plv.  riie*  obje'ci  iv<'  is  mon' ciose-lv  associale-d  with  i  In- di'sire'd 
informatif)!!  usually  |>ut  on  the'  Soar  geeal  than  the*  tie  lual  Soar  goal.  I'he  name'  objective'  was  |)ie  ke'el 
le)  avoid  confusieen  with  the'  Seear  use'  e>f  the'  weuei  getal.  Se'parating  the-  Seear  'geeal  ftemi  the'  eibje'e  tive' 
alleeWs  the*  e>b  jl'ct  i  ve''s  pe'rsisi  e'liee'  let  elilfe-r  freem  the'  Seear  'geeal's  .1 1  <h  it  e'e  I  n  ra  II V  de'hne'd  pe'isisl  egc  e'. 


Operator  cyclic-toss  <object> 

Proposal  - 

Problem  space  is  Juggling 

<object>  is  jugglabie 

The  location  of  <object>  is  in 
<hancl> 

<other-h2mcl>  is  a  hand 

<other-hand>  is  not  <hand>. 

Application  - 

If  The  location  of  <object>  is  in 
<other-hand> 

Then  The  intermediate  state  is 
noted  on  the  objective 

Objective  -  Toss-twice 

The  location  of  <object>  is  in 
<hand> 

The  intermediate  state  is  noted. 


Operator  toss  <ubjert>  from 
<hand> 

Proposal  - 

Problem  space  is  rtissing 
<objeot>  is  jugglabie 
The  location  of  <t>bje<  i>  \s  in 
<  any- hand  > 

<any-hand>  is  a  hand 
<other-liand>  is  a  hand 
<other-han<l>  is  not  <lKmd> 

Preconditions  - 

The  location  of  <ol>jert>  is  in 
<hand> 

Application  > 

If  The  name  «»f  <hand>  is  light 
Then  ross-righi  <«>bjrrt>  i<» 
<ollier-hand> 

If  The  name  of  <haii<i>  is  left 
Then  Toss-left  <objer(>  to 
<other-  haiKl> 

Objective  -  throw- to-4»iher-liaiul 

The  location  itf  the  <«»l)ject>  is 
not  in  <han<l>. 

I  he  status  of  <4>l»je<*i>  is  moving 
towaivl  <o(h<‘r-hand> 


Operator  «aich  <tvhj»*‘  t>  with 
<hand> 

Proposal  - 

Problem  s)>ace  is  tossing 
<«»bject>  is  'tigglable 
I  he  status  <jf  <objet  i  >  is  moving 
toward  <lian<i> 

Prt^conditiofis  • 

1  he  lo(-ali«>n  of  the  <ohjert>  is 
not  in  <haii<l> 

I  he  location  of  ilu*  <ohjet  t  ^  is 
(  li>se-i«>  <ljan<l> 

Application  - 

If  I  lie  iiaiiic  of  <lian(l>  is  right 
Tlu'li  (  at ch-i  igiit  <..hjc4  t  > 

If  I  he  name  of  <hand>  is  left 
Then  C  ali  h-l<*ft  <(»hj(  ct> 

(>hji>ctiv<»  -  (  atc/i' wit h-Ziand 

The  local iim  of  the  <«»hject>  is  in 
<liand> 

I  lie  status  of  <objecl>  is  hcl<l 


Prefer  the  liand  that  has  the  (»l)jec( 
Rule  - 

If  Two  Toss  operators  are 
possible  one  for  each  hand 
Then  prefer  the  operator  for  the 
liand  that  lias  the  bean  hag 


Iiiipleiiiontation  luggliiig  <'onsists  of  Tossing 
Rule  - 

If  An  linpiisse  <M  curre<l  for  an  *»|>eial«)r  <loiiig  a  task  in  tin* 
juggling  problem  space 

and  all  the  prei-onditions  h>r  tlu*  operator  are  true 
Then  Try  the  Tossing  problem  space. 


Figure  1:  Knowledge  for  eve  lie  toss  task 


To  achieve  this  persistence,  ol)jectives  ar<‘  recorded  on  the  stat('  and  can  l)e  shared.  One  major 
issue  with  the  recording  of  ohjectives  on  the  state  is  the  removal  of  I  In-se  ob  jective's.  Ifeiuovai  can 
be  (lone  with  specialized  productions  tliat  recognize  when  an  objective  lias  been  ai  lii('v<'d  and  then 
remove  the  achieved  objective,  d'liese  removal  productions  are  not  shown. 

VVe  have  left  some  knowledge  out  of  Fignri'  I  iK'canse  it  is  not  re<|nire(l  lor  onr  disenssion. 
The  key  piece  of  knowledge  left  out  is  the  operator  snbgoaling  knowledge  of  how  ob ji'ct ivc's  of 
one  operator  become  the  desiri'd  olijectivc's  for  other  o|)erators.  \\V  also  assinm'  ix'ifect  action 
p.xecution  in  this  section,  rhus  the  Toss  action  rcxpiests  reliably  loss  and  the  Calcli  action  n'(|U('sts 
reliably  catch.  We  relax  this  assumption  in  .Section  :l.  Flic  oilii'r  di'lails  of  i  Ik'  kiiowh'dgc  shown 
in  Figure  1  will  be  explained  when  we  describe  the  cxecnlion  of  the  <  vclic  toss  task. 


2.2  Task  Execution 


Fignn'  2  shows  the  execution  of  the  knowledge  in  Figure  1  along  with  som<>  siihgoalinu  ami  conllict 
resolution  knowledge.  We  will  use  it  as  an  example  of  learning  an  opi'ialor  apitlicaiion.  l  lie  top 
part  of  Figtire  2  consists  of  several  siia|)sliols  of  the  modeFs  slate  across  little.  The  hot  tom  part 
of  Figure  2  descrilx's  the  problem  solving  that  the  agent  do(>s  to  achic've  I  lie  cvclii  loss  op<'ralor. 
with  ('ach  step  marked  with  a  letter  in  a  circle.  .\.s  shown  in  the  cv(  lie  toss  opc'ialor  in  f  igure  1. 
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Initial  State  Intermediate  State-1  Intermediate  State-2  intermediate  State-3 


Problem  Space 

I  name  of  task) 


Final  (Goal)  State 


Juggling 


Select:  Cvciic-Toss 
Right  Hund 
^  Impaxse 

© 


© 


Select:  Toss  Apply:  Toss  Select:  Catch 
Right  Hand  Request;  Left  Hand 
Toss-nght 


© 


Apply:  Cyclic -Toss  ipartul) 
Note  Imermcdiaic  State 


©  © 
Apply:  Catch  Select:  Toss 
Request:  Lett  Hand 

Catchlett 


©  ® 
Apply:  Toss  Select:  Caieh 
Request;  Right  Hand 
Toss-ieli 


©  (0) 
Apply:  Catch  Task 
Request  Termination: 

Calch-nghi  Success 


Time 

Figure  2:  Execution  of  cyclic  toss  task 


the  agent  initially  knows  that  an  intemiediate  state  (where  the  Ix'an  biig  is  in  tin'  non-initial  liiunl) 
has  to  occur  and  he  noted.  But  the  agent  doesn't  ktiow  any  more  of  tin’  ilf'tails  of  how  to  do  a 
cyclic  toss.  Thus,  once  the  cyclic-toss  operator  is  selected  (B)an  iinpassi'  occurs,  fin'  |)rol)l('iii  space 
chosen  to  resolve  this  impasse  is  the  Tossitig  (irohlem  space  Ix'caiise  of  Figui(>  I's  rule  that  .liiggliiig 
is  implemented  by  Tossing  ©l  In  this  problem  space,  two  toss  oiierators  art'  i)roi)osed;  to  toss  from 
the  right  hand  and  to  toss  from  tlie  left  hand.  Flie  toss  (.perator  for  tin'  right  hand  is  s('l('ct('d  (B). 
because  Figure  I's  "prefer"  knowledge  prefers  the  hand  that  has  the  Ix'aii  hag.  Tin'  toss  ojierator 
applies,  requesting  tlie  bean  bag  be  tossed  to  the  left  hand  Fv('n  Ix'Ioit' t  Ih'  Ix'aii  bag  g('ts  to  the 
left  hand  the  catch  operator  is  select<'d  as  ap()ro|)riat('  for  catching  t  Ix'  Ix'an  hag  in  tin'  h'ft  hand  (?'. 
Once  the  bean  bag  is  caught  the  original  cyclic-toss  operator  iiotes  I  In'  inti'rmediale  stali'ijj).  ami 
again  we  have  two  toss  operators  proposcxl.  I'ln*  toss  for  ilm  left  hand  is  s('l('ci('d  (T).  and  a|>pli('s  O'. 
.\gain  the  toss  enables  the  catch  ojx'rator  to  be  proposi'd.  .iml  it  is  seh'cic'd  tis).  ami  wln'ii  the  beau 
hag  reaches  the  right  hand  it  a|)[)lies  (BX  Tin'  tossing  task  is  now  noted  as  being  acbi('V('d  b('cans('  :dl 
t  he  objectives  of  the  cyclic-toss  o[)erator  have  now  Ix'eii  aclii<'ved  (\t).  I  In'  knowl<'dg('  iliai  d<'scribes 
this  implementation  of  the  cyclic-toss  operator  (do  t()s.s  from  right,  tlien  toss  trom  h'lt  )  is  adib'd  to 
the  .Juggling  problem  space  so  that  in  similar  applications  ol  tin'  cyclic-toss  (ii)er;iioi'  an  imp;iss(' 
will  not  occur. 


Implementation  Problems 

In  the  bean  bag  tossing  probh'iti  of  Figure  2  we  selected  tin'  ioss('s  by  using  l  igiiK'  I  s  "pia'b'i  ' 
knowledg*'.  If  that  knowh'dge  was  reinova'd  from  the  system,  tln'ii  some  nn'iinxi  would  liav('  to  Ix' 
ns(>d  to  pick  Ix'tween  tin'  toss  (right)  and  toss  (h'lt)  operators  at  the  two  dei  ision  points,  f  igure  :! 
shows  what  hapfiens  wheti  the  toss  (h'lt)  operator  is  pickr'd  lirsl.  p('rli.i|)s  b('cans('  im-ans  ends 
atiaivsis  was  used  and  the  toss  (left)  operator  resolves  the  last  st('|)  in  aclii('\'ing  I  In'  (  vclic  toss. 


Figuro  3:  CnnUion  of  a  iu)n-(oiiU>in|)()raii('<)us  climik 

The  problem  solving  in  Figure  '■]  follows  that  of  Figure  2.  but  at  (p)  the  loss  (left )  operator  is 
picked  first.  Note  that  the  toss  (left)  operator  has  a  precoiulition  that  t  lie  bean  liag  is  in  tin'  left 
hand.  This  precondition  must  be  true  before  it  can  apply.  In  Figure  ;}  ibis  precondition  causes 
an  impasse  and  operator  subgoaling  knowledge  seleits  the  Possing  iiroblem  sjiaci'  to  ri'solvi’  the 
impasse  with  the  goal  being  to  get  the  bean  bag  to  the  left  hand  (K).  I’he  toss  (right )  ojx'rator  is 
selected  0and  applies  (<]) in  this  lowest  (iroblem  space.  Since  the  bean  bag  is  now  moving,  the  catch 
operator  becomes  selectable,  and  is  selected  (jj).  Sometim*'  lali'r.  tin*  bean  bag  is  close  luiough  to 
the  hand  for  the  catch  operator  to  apply  0.  The  catch  operator  makes  the  bean  bag  be  in  tin'  left 
hand  resolving  the  task  for  the  lowest  iiroblem  s()ac<' (t),  and  allowing  both  tin'  cyclic-toss  operator 
to  note  the  intermediate  state  0.  and  the  toss  (left)  operator  to  apply  (£).  .\s  In'fore.  tin'  In'an  bag 
is  caught  by  the  right  hand,  the  Tossing  task  is  resolved  (o)and  the  cyclic-loss  is  learin'd. 

To  an  external  observer,  the  scenarios  in  Figurr*  2  ami  Figure  ;{  are  tin'  same,  ainl  the  knowh'dge 
that  resolves  the  cyclic-toss  operator  is  simply  to  first  toss  from  tin'  right  hand  and  l  In'ii  toss  from  t  he 
left.  I’nfortunately,  the  chunk  created  by  Soar  for  t  ln>  cyclir -loss  Soar  operator  from  t  he  applical  ion 
of  the  catch  (left)  Soar  operator  in  Figure  2  is  ina|)|)licable.  I'he  cri'ation  of  tin'  loss  ( h'fi )  opi'iator 
de[)en(ls  upon  the  bean  bag  being  held  in  a  hand,  se*-  Figure  1,  II  tin'  Ix'an  bag  was  not  In'ing 
held,  then  perhaps,  a  different  operator  would  be  propose<l  to  axhii've  tin'  stati'  of  holding  tin'  bean 
bag.  Testing  that  the  bean  bag  is  being  held  makes  the  loss  (left)  o|)('rator  in  I’igiire  .'i  depemh'iil 
upon  the  bean  l)ag  being  in  some  hand.  When  the  bean  bag  is  losst'd  from  tin'  right  hand,  this 
dependency  still  exists,  but  it  is  in)  longer  true.  We  havt'  a  Ir'inporally  inconsislc'iit  situation  with 
r*'gards  to  the  toss  (left)  operator.  So  wln'ii  tin*  application  of  tin'  catch  (left)  o|)eralor  occurs, 
another  chunk  is  built  that  lncf)rporates  the  I'atch  (left)  action  rr'cpn'sl  ainl  tin'  <l('[n'nd<'ncy  that 
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If  The  prolilein  space  is  Juggliiiff 
the  operator  is  cyclic-toss 
the  <object>  is  jugglable 
the  <object>  is  moving  toward  the  <haml> 
the  location  of  the  <object>  is  not  in  <haiid> 
the  location  of  the  <objecl>  is  in  < any- hand > 

<  any- hand  >  is  a  hand 

the  location  of  the  <object>  is  close-to  <hand> 
the  name  of  the  <hand>  is  left 
Then  catch-left  <object> 

Figure  4;  Example  of  non-conlemporaneoiis  knowledge 

If  The  problem  space  is  .Juggling 
the  operator  is  cyclic-toss 
the  <object>  is  jugglable 
the  <obJect>  is  moving  toward  the  <hand> 
the  location  of  the  <object>  is  not  in  <hand> 
the  location  of  the  <object>  is  close-to  <hand> 
the  name  of  the  <hand>  is  left 
Then  catch-left  <object> 

Figure  -j:  Example  of  useful  knowledge  created  on  s(>cond  attcmipi. 

the  bean  bag  is  in  some  hand.  Thus  the  catch  (left)  is  dependent  both  upon  tin'  bean  hag  Initially 
being  held  by  a  hand,  from  tlie  toss  (left)  operator  creation,  and  being  moving  and  close  to  the  h'fi 
hand  at  the  time  of  the  catch  from  the  catch's  preconditions.  However,  the  temitoral  aspect  of  this 
scenario  is  lost.  The  chunk  generated  for  the  catch,  as  shown  in  Figure  4.  re(|uir('s  that  tin’  bean 
bag  be  both  moving  and  in  a  different  hand  than  the  left  (catching)  hand.  This  is  hardly  a  good 
situation  for  catching  a  tossed  bean  bag. 

The  chunk  in  Figure  4  is  an  instance  of  a  non-contt  mpomutous  clntiifc  (see  [Laird  and  Huirtiian. 
1992]).  It  is  splitting  the  cyclic-toss  operator  into  th<*  two  ()orfions.  The  first  portion  cxc'ciites 
before  the  chunk  doesn't  apply,  making  the  first  toss  of  the  Ix'an  bag.  LIk'  scmoikI  <ati't  applv 
because  the  bean  bag  is  not  caught. 

The  main  issue  here  is  that  the  cyclic-toss  operator  should  tiot  care  whet  In'r  the  method  shown 
in  Figure  2  or  in  Figure  4  was  used  for  solving  the  cyclic  toss  problem.  I'he  chunk  shown  in  Figure  I 
will  not  apply  at  the  appropriate  time  becau.se  the  conditions  will  not  match  any  Juggling  situation. 
Thus  the  knowledge  transfer  for  the  cyclic-toss  operator  failed  to  generali'  efh'ctivc'  knowledge.  This 
failure  to  apply  causes  the  cyclic-toss  operator  in  one  problem  solving  sctuiario.  Figure  to  not  be 
as  effective  as  in  the  other.  Figure  2.  even  though  no  a  priori  reason  (exists  to  prc'h'r  one  ov(u  t  h(' 
other. 


Recovery  from  split 


The  situation  in  Figure  4  is  not  as  bad  as  it  might  s('em  b('caiis('  t  h('  |)roper  knowledg('  is 
built  when  the  cyclic  to.ss  is  next  attempted.  Fliis  happens  because  the'  (  vclic-toss  op('rator  was 
implemented  with  simpler  Soar  operators  that  worked  (uttirely  basc'd  on  the  situation  in  the  top 
problem  s()ace.  When  t  he  cyclic-toss  operator  is  selected  sometime  in  the  future,  and  t  lu'  Ix'an  bag 
is  in  the  right  hand,  the  first  chunk  corrc'sponding  to  throwing  the  luuin  bag  from  the  light  iiand 


Figure  (j:  C>raph  of  operator  goal-<le|)t  li  l)y  Icuuning  trial 

will  apply,  sending  the  bean  hag  to  the  left.  hand.  The  s('(()nd  <hunk  for  catching  tin'  Ixmii  hag 
in  the  left  hand  can’t  apply  because  it  requires  the  b<>an  l);ig  to  be  in  the  right  hand  and  it  iMi’t. 
Thu.s.  an  impasse  will  occur  because  the  cyclic-toss  hasn't  coni|)i('t<'d.  Iti  r('solvitig  this  ittii)ah>e.  the 
Tossing  problem  space  is  picked  again.  Sitice  tiie  current  situation  tiiittches  the  itroposal  cotidiiions 
for  the  catch  operator,  it  will  Ite  proposed  and  applied,  catchitig  the  luntii  bag.  I'liis  time,  however, 
the  catching  reqttest  creates  the  useful  chutik.  showti  in  Figun*  a.  for  th('  Soar  cyclic-toss  op('tator. 
Thus,  in  this  case  So;ir  learns  the  apjtropriati'  knowl('dg<>  ovc'r  multiple  tries  rather  fh.an  sintplv 
from  the  initial  learning  situatioit. 

The  strategy  tis('d  in  Figitre  2  ;iitd  Figure  .{  for  creating  Soar  macro  operators  sijch  as  cycli<  -ioss 
is  very  powerful.  If  the  simple  Soar  o|)erators  that  deliiu’  the  Soar  macro  operatois  I'itin'r  aUsavs 
work  in  the  Stjar  stat('  that  the  Soar  macro  opcuatoi  is  working  in.  or  can  always  recreate'  any 
local  state  information,  then  the  (rrogram  can  always  recover  from  non-conti’inpoi ain'ons  chnnks. 
This  is  important  because  it  ('usures  an  e'lfective  imirleim'iitat  ion  of  t  hi’  I’St'M  o])erat  or.  e\-eii  if  the 
problem  solving  in  Soar  is  more  complex  than  in  the  PSCM  model. 

Figure  b  shows  the  iti’rativi'  gi’ueration  of  increasingly  general  chunks  in  a  more  comirlicated 
task  of  a  robot  juishing  a  box  (descriited  further  in  Section  1 1.  Fhe  X  axis  of  Fignie  ti  counts  the 
operators  reipiired  to  do  the  task  in  each  t  rial.  The  axis  is  t  he  di’pt  h  oft  he  goal  si  ack  that  existed 
when  the  operator  was  proposi’d.  Four  consecutive  learidng  trials  on  the  s.ime  problem  are  shown, 
riie  problem  solving  in  the  first  trial  creates  some  non-conlem|)oraneons  chunks.  I  liese  chunks  <lo 
not  fire  in  the  second  trial,  causing  im|)asses  to  resolve  the  missing  kno\sledge.  The  new  chunks 
from  the  second  trial  are  useful  up  to  the  point  that  they.  too.  become  non  i  onteniiioraneous.  This 
process  repeats  itself  until  we  have  usi'ful  immediate  know|e<|g<>  available  at  the  end  of  trial  -f  ,ind 
do  not  need  to  impasse  in  trial  1. 

Failing  to  recover 

When  a  non-contemporaneous  chunk  fails  to  ap|)ly.  any  other  knowledge  depmnlent  on  its  ai  lions 
will  also  fail  to  apply,  ifecovering  is  not  necessarily  as  easv  as  described  above.  The  dillicnit 
'it  na  I  ions  occur  in  operator  a  pplicai  ion.  when  a  jiort  ion  of  I  he  opera  I  or  a  pplies.  thus  making  some 
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Figure  7:  Cyclir  toss  witli  syiiimetric  toss-<al(ii  oix'iators 

previous  portion  of  the  operator  iiiapplicaltle.  If  the  now  ina|)pli<al)l('  ixulioii  of  iht'  opt'rator  is 
required  for  recovery  then  a  problem  exists.  VVe  ran  create  such  a  situation  by  modifying  our 
example,  so  that  instead  of  having  toss  ainl  catch  opc'rators  in  tin'  Possiug  jiiobhun  space.  w('  I  rv 
to  implement  the  cyclic-toss  operator  with  the  symmetric  toss-left-ciitch-right  iuid  toss-righl-catch- 
left  operators  in  the  Throwing  problem  space.  VVe  can  assume  that  the  I’lirowiiig  liioblem  space 
operators  were  learned  from  the  Tossing  problem  space  operators  some  time  iu  the  past.  I'lius  w(' 
learned  how  to  throw  from  hand  to  hand  before  attempting  the  cyclic  toss.  I'lifort  uiiati'ly.  ri'covery 
with  knowledge  of  this  form  only  becomes  po.ssible  if  we  expand  o)ir  current  imu  liods. 

We  will  explain  why  recovery  is  difficult  by  going  througli  our  same  (  vclic  toss  example.  VVe 
are  assuming  that  the  proposal  conditions  of  these  comirh'X  toss  and  catch  opi'rators  art'  the  same 
as  those  for  the  toss  operators  of  Figure  1.  and  that  the  catch  action  occurs  wlnui  tin'  catch 
preconditions  are  true.  Figure  7  shows  the  cyclic  toss  being  doin'  with  th('s('  oix'ialors  assuming 
the  left  hand  is  tried  first.  When  the  left  hand  is  tried  (fi),  the  same  im|)ass('  from  Figur*'  d  of  an 
unresolved  precondition  occurs  and  leads  to  the  toss-right-catch-h'l't  oix-rator  being  picked  0  in 
a  new  Throwing  problem  space.  'Phis  operator  applies  tossing  the  Ix'an  bag  to  tin'  h'jt  hand  0 
and  catching  it  in  the  left  hand  Once  caught,  tiu*  bottom  throwing  task  is  l('rminate(l  0.  the 
intermediate  state  is  noted  0,  and  the  loss-left-catch-right  operator  can  now  start  afiplying.  It 
requests  the  toss-left  action  0and  then  the  catch-right  action  0.  .Vgain  wIk’ii  I  Ik'  Ix'an  bag  is  back 
in  the  right  band,  the  task  terminates  I’lie  learning  is  very  similar  to  that  in  Figuri'  •{.  and  a 
non-contemporaneous  cliniik  is  created  for  the  cyclic-toss  oix'iator  lor  t  h('  calch  h'ft  action.  Phis 
chunk  looks  exactly  like  our  previous  chunk  in  Figure  I. 


On  the  next  attempt  at  a  cyclic  toss  that  starts  from  the  right  hand,  the  first  cliunk  for  the 
cyclic-toss  operator  applies;  tossing  the  bean  bag  from  the  right  hand.  Since  the  second  chunk  can’t 
apply,  an  impasse  occurs.  The  Throwing  problem  space  is  picked,  but  no  operators  are  ])roj)osed. 
Both  toss-left-catch-right  and  toss-right-catch-left  operators  are  only  proposed  if  the  bean  l)ag  is  in 
a  hand.  Since  the  bean  bag  is  in  the  air.  no  operators  are  proposed.  This  restdts  in  a  new  impasse, 
but  we  have  no  knowledge  relating  to  the  handling  of  this  impasse,  so  the  l)ean  bag  falls  lo  (he 
floor. 

Not  being  able  to  independently  catch  the  flying  bean  bag  in  the  Throwing  prol)lem  space  is  the 
root  problem  in  this  e.xample.  The  knowledge  about  catching  in  the  left  hand  was  available  in  the 
Throwing  problem  space,  it  was  just  embedded  in  the  wrong  operator.  Even  if  the  rossing  problem 
space  with  our  original  toss  and  catch  operators  was  available  to  resolve  the  impasst'.  we  could  not 
have  recovered,  because  the  issue  to  be  resolved  by  this  im|)a.sse  is  that  no  operator  is  available  in 
the  throwing  problem  space.  To  recover,  somehow  we  have  to  extract  ilu'  catching  knowh'dge  and 
make  it  available  in  the  Throwing  problem  space  in  the  form  of  a  new  operator. 


2.3  Generalizing  Recovery  Methods 

This  section  describes  how  to  write  Soar  models  that  exhibit  tin'  recovf'iy  capability  we  >hw 
in  the  previous  section,  while  avoiding  the  problems.  In  i>articnlar.  it  is  coiici'ined  with  non- 
contemporaneous  chunks  that  arise  out  of  interaction  with  theexternal  world,  .\’on■contemporaneou^ 
chunks  can  also  be  created  by  strictly  internal  problem  solving,  and  the  ■-ame  methods  help  in  these 
situations,  especially  if  the  problem  stems  from  |)lanning  and  using  knowledgi'  about  the  ('xtenial 
world  responses. 

Recovery  consists  of  two  parts:  reronslrui'limj  tfn  roiilat  of  the  probh'm  solving  up  to  tin' 
point  it  was  disrupted  (i.e.  when  the  non-coulemporaneous  chunk  didn’t  lire)  and  miiliiiiintfi  Ihi 
problem  solving  after  the  disruption.  When  the  learned  application  of  the  cyclic-toss  operator 
was  disrupted  by  the  non-application  of  the  non-contemporaneous  chunk  in  Section  2.2’s  posiiivi' 
e.xample,  the  problem  solving  continued  using  the  original  problem  solving  knowh'dge.  In  gen¬ 
eral.  non-contemporaneous  chunks  disrupt  the  immediat*'  use  of  o|>erator  application  knowledge. 
The  disruption  camses  an  impasse  that  can  be  ii.sed  to  learn  knowledge  that  can  rep|ac«'  tin-  non 
contemporaneous  chunk,  if  the  correct  context  can  be  created.  In  Section  2.2’s  negaiivi'  examph'. 
the  context  could  not  be  created  that  would  allow  ih<'  catch  portion  of  tin'  loss-iiuht  (atch-lelt 
operator  to  apply.  The  context  reconstruction  i)art  of  recovc>ry  has  a  continuum  id  ^nliitiuns  that 
range  from  doing  nothing,  becau.se  all  the  stale  b/'nve/ui  b»»lh  /xoblem  spaces  is  sliajed.  to  com 
pletely  reconstructing  the  context  in  the  lower  problem  space.  In  the  lirst  case  the  problem  solving 
in  the  lower  problem  space  duplicates  all  its  results  in  the  ui)|)er  problem  space.  I  hus.  when  the 
non-contemporaneous  chunk  refuses  lo  fire  we  hav<>  |)recisely  the  lontext  i\eeded  to  proceed.  In 
the  second  case  we  keep  no  intermediate  results  but  insteaii  have  a  method  for  rec oust rui  t ing  the 
problem  solving  whenever  it  is  neederl.  We  now  rh'sr'ribe  two  example  systems  that  can  be  classilii'd 
as  different  points  along  this  continuum. 


Sharing  state 


riie  cyclic-toss  operator  of  Figure  .{  is  au  examph'  of  sharing  the  ma  jor  problem  solving  results  ol 
the  |)roblem  solving  in  a  lower  problem  space  with  the  u[)per  problem  spaie.  .\  shared  portion  ol 
the  state  holds  the  problem  solving  context.  In  Figure  d  the  shared  problem  solving  loniexl  is  the 
location  and  status  of  the  bean  bag,  and  the  nhjd-lin  of  (he  cyclic-loss  operator.  This  e\am|)h'  only 
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requires  the  perceptually  iletermiiied  location  and  status  of  the  bean  l)ag.  When  tin'  impasse  occurs 
in  the  cyclic-toss  operator  the  second  time,  the  bean  bag's  status  of  “moving"  is  what  enabh’s  tin* 
catch  operator's  selection  and  thus  the  learning  of  the  correct  knowledge.  In  Figure  7  the  problem 
solving  context  for  each  of  the  catch  actions  includes  the  fact  that  the  associated  toss  was  done. 
Thus  in  this  case,  all  the  context  is  not  shared,  the  fact  that  the  associated  toss  was  done  is  "known" 
only  in  the  sub-context.  Thus,  when  the  impasse  occurs  after  the  toss,  the  system  can't  continue 
because  it  can't  recreate  this  specific  context  for  the  catch  action. 

In  Figure  3  the  objective  does  not  need  to  be  shared,  to  solve  this  particular  problem.  I'he 
sharing  is  done  for  demonstrative  purposes,  since  sharing  of  the  objective  is  r<>(|uired  in  more 
comple.x  problems.  In  some  more  complicated  systems  objectives  are  linkofl  to  implement  a  -^tack 
mechanism  to  focus  the  effort. 


Replanning 

.\t  the  other  end  of  the  spectrum,  the  reconstruction  process  can  re-derive  the  context  information 
as  needed.  Consider  Mitchell's  robot,  that  has  the  multiple  goals  of  finding  a  cu])  and  keeping  itself 
charged  [Mitchell,  1990].  The  robot  determines  an  action  by  planning  and  then  re(|uests  the  action 
forgetting  all  the  planning  knowledge  involved  in  picking  that  particular  action.  If  .Mitciu'll's  roi)oi 
didn't  learn,  it  would  completely  replan  for  each  action  reciuest.  .Mitchell  makes  this  r<'plannim> 
efficient  by  having  the  robot  cache  the  planning  results  as  stimulus-response  rules.  I'lie  conditions  of 
the  rules  are  generated  through  an  explanation-ba.sed  generalization  of  tlu'  original  planning.  I'lius. 
the  next  time  a  similar  situation  arises,  the  cached  rule  will  fire  providing  the  action  rtuiuest  and 
make  replanning  unnecessary.  What  Mitchell  does  not  do  is  cache  intermediat(>  planning  results, 
only  the  final  ones. 

.Mitchell’s  robot  is  an  example  of  rederiving  the  context  information  when  it  is  nec'di'd.  Figitn' s 
shows  one  possible  l’S(,'.\I  version  of  .Mitchell's  method  learning  tlu*  cyclic  toss.  Tlu*  ('xt'ciilioti 
trace  on  the  left  shows  the  planning  that  initially  occurs  and  the  execution  tract*  on  tlu*  right  t  he 
application  of  the  learned  rules.  In  Figure  X  the  operators  are  shown  in  bold,  flu*  action  r<'(|U('st.^ 
and  any  state  changes  ajrpear  in  normal  roman  font,  atui  indentation  iudicat(*s  proc(*ssing  within 
the  operator.  The  text  in  sans  serif  font,  is  the  planning  activity.  (’hang(*s  imuh*  to  tlu*  state*  in 
the  sans  serif  section  are  removed  when  the  action  retpiest  is  actually  made.  If  an  op(*rator  in  curs 
within  another  operator,  the  sub-operator  must  be  in  the  context  of  a  sub-prol)h*m  s|)ac('. 

Comparing  the  left  hand  side  of  Figure  S  to  tlu*  cyclic-toss  (*xample  in  Figure*  J,  MitchelFs  te)be)t 
eloes  more  planning  to  eleterrnine  that  the  te)ss  (right)  actieui  ree|ue’st  will  le*ail  towarel  the*  obje'e  t ive*. 
When  the  toss  (right)  action  re*eiue*st  is  maele  a  rule  is  cre*ali'el  that  will  re'e|ue*st  a  similar  aetioii 
in  a  similar  situatiejii.  .All  the  rule’s  createel  in  Figure*  X  are*  ve'ry  similar  te)  the*  ehunks  cre'ateel 
in  Figure  2.  except  that  the  interme>eliale  state  information  is  inclueleel  in  all  the*  ehunks.  riu* 
intermeeliate  state  information  is  inclueleel  be’cause  a  eeuuplete*  plan  is  use'el  te)  elete'rmine*  tlu*  actions 
and  in  the  eomplete*  j)lan  the  interme*elia.te*  state*  value  is  impe)rtant.  Froei'e*eling  eleewn  tlu*  le'fl  hanel 
siele  e)f  Figure  X.  when  the*  lu-an  bag  le*ave*s  the  right  hanel  me)ving  te)warel  the*  h’ll  hanel.  the*  lereeei-ss 
of  eletermining  tlu*  next  actiejii  starts  anew  from  the  cyclic  te)ss  ge>al  anel  the*  pe-ree'pt ual  kne)wle'elge' 
that  tlu*  beat)  bag  is  moving  towarel  the*  left  hanel.  Tlie  jtrocess  rebuiUls  whate*ve'r  eetiile'xt  is  ne'e'eleel 
te)  eleterinine  that  the  catch  (left )  ae  tie)n  is  t  he  lu-xt  actieui  that  leaels  towarel  I  he*  e)bje'e  t  ive*  anel  I  bus 
slu)ulel  ne)W  be  re*eiueste*el.  .A  ne*w  rule*  for  this  plan  is  netw  ereate'el.  I'liis  pre)ee*ss  1^)11!  iuiu’s  until 
tlu*  cyclir  toss  objective  is  reacheel.  riu*  right  hanel  .“iele  of  Figure  X  sluiws  the*  applieat ieen  of  the 
learneel  rules.  e)ne*  for  eaeh  eye  lie-toss  eepe'iator.  I'he*  eyclic-teess  opi'rateer  is  se'rving  as  the*  goal  in 
■Mitchell's  system. 


Initial  Problem  solving  Trace 
(before  learning) 


Final  Problem  Solving  Trace 
(after  learning) 
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Figure  S;  Mitchell's  system  -  Always  re-|)laiiniiig 

The  PS(hM  implementation  shown  in  Figure  S  dilfers  from  Milchell's  work  in  two  ways.  First 
Mitchell's  robot  doesn’t  have  operators,  meaning  that  the  stimulus-tesi)onse  iiiles  apply  immedi¬ 
ately  without  any  decision  process  occurring.  The  only  issii<*  this  raist's  is  that  soiik'  |■easolli^g 
may  be  occurring  in  Figure  S  (i.e.  in  Soar)  betwemi  the  cn'alion  of  llu*  opt'rators.  ami  tin'  s<'l('c- 
lion  of  the  operators.  .Mitchell’s  system  encapsulates  this  reasoning  in  the  left-hand  side  of  the 
stimulus-response  rule.  Such  an  encapsulation  iussumes  that  no  new  goals  or  knowh'dgi'  will  change' 
the  reasoning  process.  The  second  difference  between  Figure  S  and  MitclK'll's  work  is  that  action 
requests  are  idempotent.  that  is  they  can  be  re<|uested  multi|)le  times  and  have  the  same  elfecl  as  if 
they  wore  reepiested  only  once.  Our  action  reepiests  recognize  the  action  has  already  be<'n  r('(|uesl('d 
and  uses  the  recognition  to  withhold  the  same  action  b«'ing  request.c'd. 

.Mitchell  avoids  non-contemporaneous  stimulus-respon.se  rules  by  working  IVotn  a  snap-shot  of 
the  world,  having  the  only  result  be  the  action  r«'(|U('st,  and  always  r<'-planning.  Siuc('  h<'  works 
from  a  snapshot,  the  e.xternal  world  cannot  change  while  he  is  doing  the  planning.  <'liminatitig  on*’ 
source  of  non-cotitemporaneous  stimulus- respon.se  rules.  .Since  he  has  no  p»'rsist<'nt  objt'cts  in  his 
planning  sub-goals  (he  always  re-derives  the  objects  from  iIk'  new  snap-shot),  he  has  ('liminaled 
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the  other  source  of  aoii-contemporaueous  stimulus-response  rules.  Soar  model  can  always  do 
the  re-planning,  but,  as  we  will  show  in  Section  4.2,  this  has  implications  for  the  number  of  Soar 
operators  required  to  achieve  an  objective.  A  Soar  model  can  also  specify  that  all  planning  is  done 
in  a  snap-shot  of  the  external  world,  though  it  is  not  usually  done. 


Making  recovery  work 

Once  the  problem  solving  context  is  restored,  then  the  original  knowledge  can  automatically  apply 
and  the  problem  solving  continues.  Creating  a  system  so  that  this  process  can  occur  easily  is  more 
an  issue  of  context  reconstruction  than  problem  solving  continuing.  VVe  have  shown  (wo  ways  to 
reconstruct  the  context,  both  of  them  benefiting  from  the  u.se  of  small  o|)era.lors  at  the  base  of  the 
operator  recursion.  The  smallest  operators  are  tho.se  that  do  one  change  to  the  state,  and  then 
terminate.  Building  productions  from  the  small  operators  means  that  when  an  imi)asse  occurs  in 
the  application  of  a  complex  operator,  other  operators  can  apply.  Unfortunately,  this  only  works 
if  the  small  operators  are  available  in  the  problem  space  that  re.solves  the  impa.sse.  This  section 
describes  a  method  to  make  the  small  operators,  and  thus  their  knowledge,  available  again,  through 
the  creation  of  a  new  operator.  The  method  is  not  .automatic  today,  but  it  shows  how  a  Soar  model 
can  be  changed  to  work  around  this  problem. 

The  lack  of  an  operator  arises  because  the  impasse  can  occur  in  the  middle  of  the  complex 
operator’s  implementation.  .Vlso.  the  impassed  situation  might  not  match  any  of  the  sub  problem 
space's  operator's  proposal  conditions.  VVe  .saw  this  in  Figure  7  aft(>r  tin'  initial  h'arning.  The 
cyclic-toss  operator  impassed  after  the  first  toss  becau.se  the  chunk  to  do  the  catch  didn't  ap|)ly. 
The  Throwing  problem  space  was  ,selecte<l  to  resolve  the  impa.sse.  but  the  proposal  conditions  of 
the  two  operators  in  the  Throwing  problem  space  didn't  match  the  state  of  th('  bean  i)ag  in'ing  in 
the  air. 

One  way  to  ensure  that  the  small  operators  are  available  is  to  do  all  the  work  within  a  single 
problem  space.  However,  making  all  the  operators  available  in  one  huge  probh'in  space  precludes 
problem  spaces'  organizational  .advantages.  .Vl.so.  the  existence  of  multiple  i>robh'm  spaces  is  not 
the  issue,  we  assume  the  complex  operator's  implementation  was  learned  by  a  previous  trav('rsal  of 
the  problem  space  structure.  The  creation  erf  a.  new  operator  will  h  rc('  the  problem  space  structure' 
to  be  traversed  again,  relearning  the  [rarticular  knowle'elge  that  is  a|)|)re)priate'  tee  this  ne'w  situatieui. 
This  new  operator  will  .also  encapsulate  this  new  knowle'elge. 

Creating  another  operator  could  be*  done  autemiatically  in  (he*  imirasse'  that  oce  urs  be'eause'  uer 
operator  is  available,  but  care  has  to  be  maele  ser  that  the*  prerpeesal  eeuidit iems  of  the'  ue'w  ope'raterr 
are  reasonable.  In  this  situation,  we  dern't  want  to  learn  ter  e  ate  h  just  any  ob  je'ct.  ordy  t  hose'  objects 
thrown  from  the  other  hand.  How  to  automatically  create'  the*  erperator  is  be'vond  (In'  scope*  of  this 
paper.  The  robot  work  describenl  here  shares  a  common  [rrobh'm  space'  and  thus  doe's  not  have'  this 
particular  problem.  The  work  that  has  been  done  in  this  are'a  has  cre'ate'd  overly  s|)e'cilic  chunks 
that  were  linked  to  the  particular  obje'ctives  of  the  cydic-toss  e)|)erator. 

• 

The  reason  for  creating  a  new  ojrerator  is  that  the  mexlel  made  a  commit  meut  to  a  part  icidar  set 
of  knowledge  in  a  particular  set  of  operators,  and  that  structuring  has  turue'd  out  to  be*  wrong.  It 
is  recognizing  that  the  current  organization  of  the*  knowledge  into  opc'iators  is  causing  ( In*  probh'in. 
The  inapplic.al)ility  of  knowledge  that  exists  within  a  differi'iit  ope'rator  just  me'ans  (hat  a  ne'w 
operator  has  to  be'  created  that  can  also  encompass  that  knowledge',  possibly  elu|)licating  it. 


2.4  Eager  vs.  Lazy  Detection 


So  far  we  have  been  describing  how  non-contemporaneous  cluinks  can  cause  operators  that  do 
multiple  actions  to  split.  In  essence  the  problem  is  that  the  learning  method  in  Soar  does  not 
correctly  handle  non-contemporaneous  information,  and  the  fact  that  a  temporally  inconsistent 
situation  did  e.xist  is  detected  only  when  the  chunk  fails  to  apply.  .\n  alternative  approach  is 
to  change  the  architecture  to  detect  when  objects  become  tion-contemporaneous  and  take  the 
appropriate  actions  so  that  a  non-contemporaneous  chunk  is  never  created.  Such  a  change  to 
Soar  has  been  suggested  in  a  mechanism  called  S-Su|)port  [Laird  and  Ilulfman,  1!)92].  The  main 
difference  between  a  Soar  system  without  S-support  and  one  with  S-support  is  when  the  fact 
that  non-contemporaneous  information  was  used  is  discovered.  .A  Soar  system  without  S-support 
detects  this  situation  when  an  attempt  is  made  to  use  knowledge  built  on  non-contcunporaneous 
information,  because  the  chunk  fails  to  apply.  We  call  this  la:y  detection,  because  it  is  happening 
as  late  as  possible.  The  Soar  system  with  S-support  detects  and  removes  the  non-contemporaneous 
information  even  before  it  can  be  used.  This  is  ;•  <letertion.  I)ecanse  it  is  as  earlv  as  possible. 
The  purpose  of  this  .section  is  to  show  that  early  detection  also  splits  the  operator.  re(|niring  some 
form  of  recovery  as  in  the  previous  section. 


S-support  also  splits 

First  we  note  that  learning  in  Soar  rerpiires  an  impasse  an<l.  for  tiny  impass(>.  wi>  hav<'  a  super-<'onte.\t 
(the  impassed  one)  and  a  sub-context.  The  creation  of  a  non-coniemporam'ons  chunk  dept'mls  ui)ou 
the  existence  of  at  least  one  persistent  suit-context  ol)j«'ct  generated  from  a  >nper  ( oiite.xt  object 
that  has  changed  since  the  generation.  In  the  e.xample  shown  in  Fi.gun'  tin*  persisimii  >ul)-context 
object  is  the  toss  (left)  operator  that  was  created  wlnui  the  tx'an  bag  wa.^  in  the  right  hand  b).  flic 
bean  bag’s  location  is  the  changed  siipcu-contexl  object.  S-supi)orl  is  a  proposed  (hange  id  the 
Soar  architecture  that  changes  the  persistence  rules  in  Soar  to  remove  all  sub-conte.vt  object-,  that 
were  generated  from  changed  su()<'r-cont<‘Xt  objeci.s.  Hemoval  of  the  non-contemporaneous  objects 
removes  one  of  the  conditions  ( iiersisteiice)  that  allow  non-contetnporaneous  (  hunks  to  Ix'  i  reatc'd. 
Thus,  the  Soar  jirogram  cannot  create  a  tion-c<uil<'m|)orain'on.s  chunk.  Ilowi'vi'r.  furtln'r  itrobhun 
solving  might  be  dependent  on  the  object  that  S-support  rcmiovc'd.  For  this  problimi  solving  to 
continue  a  new  similar  object  with  S-support  has  to  l>e  creatf'd.  Tin'  piDci-ss  tliai  ( reati's  the  new 
object  turns  out  to  be  the  same  process  that  tin'  recovery  nn'lhod  use's  to  n'-constrm  t  the  (ontext 
after  a  non-contemporaneous  chunk. 

■As  an  example,  in  Figure  d  the  original  toss  (left)  operator  was  di'pemh'ut  u|>on  tin-  Ix-an  bag 
being  in  a  hand  (as  seen  on  the  left  in  Figure  0).  I’lx'  right  side  of  Figure'  !)  --hows  that  wlx'ii  the- 
bean  bag  appeared  in  the  left  hand  this  toss  (le'ft)  ope'ialor  lost  one-  me'inbe-r  ol  its  su|)e'r  e ontext 
support  set.  and  thus  woidd  be  te'rminateHl  by  S-supporl.  indie ate'd  by  the'  gre\  ".X"  in  Figure  !). 
Something  that  was  not  sen'ii  in  our  simple  e'xamph'  was  that  whe'ii  tin'  be'an  bag  was  in  the'  air  a 
catch  (left)  operator  was  jiroposed  and  this  catch  (left)  ope'iator  is  de'pe'nde'iit  upon  the'  be'an  bag 
l)eing  in  the  air.  Hejwever  in  exir  ejriginal  syste'in.  since'  we'  hael  what  appe'are'el  te>  be-  a  pe'rfe'etiv 
good  f)peratcjr  already  selected  this  ne'w  (irerpeu.al  was  igne)re'd.  VN'ilh  S-support  le'rminating  the' 
errigirial  teiss  (left)  operator,  the  new  catch  (hTt)  e)pe'rator  can  be>  se'h'cte'd.  l  lins.  tin'  e'xam|)le’  iti 
Figure  ;{  with  S-support  wouhl  learn  all  the  right  chunks  in  the*  iirsi  pass,  rathe'r  than  tin'  twe> 
passe's  used  with  non-contemporane'ous  chtmks.  S-suppeui  assunx's  that  proble'ins  will  oecur  wln'ii 
a  te'tnporally  inconsistent  situation  arise's  before  any  chunks  are'  built.  .Note'  that  the'  me'lhexi  eef 
recovery  is  the  same  here  as  wIk'Ii  the  non-a.p|)licalie>n  eef  a  inin-conte'mpeuain'exis  <  Imtik  simwe'd  a 
[>a.st  temporal  iitconsiste'ncy.  In  the  e'xample>  in  Figure'  :$  hot  h  e  ase  s  have'  to  se-lect  and  .ipply  a  cate  h 
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Figure  9:  Changing  (lie  super-context  support 


(left)  operator.  Using  noti-conteinporaueous  chunks  is  lazy  l)ecause  t  h<'  new  catch  (left)  operator 
is  applied  at  the  latest  possible  time;  S-support  is  eager  because  it  applit's  it  as  soon  as  jiossible. 

The  interesting  concept  here  is  that  recovering  from  an  .S-support  retraction  of  support  is  exactly 
the  same  as  recovering  from  a  non-contemporaneous  chunk.  Tims.  S-sni)port  should  not  alfect 
programs  that  can  recover  from  non-contemporaneous  chunks,  and  programs  that  can'l  recovc'r 
from  the  loss  of  S-support  to  objects  cannot  recover  from  non-contem|)oraneons  chunks.  This 
suggests  that  S-support  does  not  change  the  set  of  elfective  Soar  programs,  but  is.  instead,  an 
implementation  mechanism  that  speeds  up  learning  by  forcing  Soar  |)rograms  to  (Imd  with  the  loss 
of  super-context  support  immediately.  The  situation  is  analogous  to  the  type  systmn  in  modern 
programming  languages  like  ML.  The  .ML  language  states  that  all  programs  will  lie  lyjx'  correct. 
Implementing  compile-time  type  checking  in  .ML  is  not  a  change  to  i  li(>  set  ol  valid  .ML  programs, 
it  is  just  an  implementation  mechanism  that  forces  one  to  deal  with  lypi*  problmns  at  ilu'  compile 
stage. 


2.5  Relationship  of  the  methods 

VVe  have  rlescribed  two  methods  (s-sui)[)ort  ami  non-contem|)oraneous  chunks)  lor  detm  ting  a  tem¬ 
porally  inconsistent  situation,  and  two  metluxls  ( r('|)lanniug  ami  sharing  state)  lor  r('coveriug  from 
the  effects  of  such  a  situation.  I'he  reasons  for  picking  a  detection  and  recovery  iinu  hod  are  inde- 
[lemlent.  and  thus  can  be  mix<‘d  depmiding  upon  the  model’s  rerpiirmiu-ut s.  I'o  sumtuarizm 

Detection  -  .Since  this  is  architecturally  dcdined  wi>  have  an  eii  her/or  situation. 

•  S-support  - 

-  Karly  detection  of  temporally  inconsistmit  situations. 

-  Faster  learning. 

-  Unnecessary  work  done  by  architecture  nunoving  itc-ms  that  will  not  be  ns<'d. 

•  .Non-contemporaneous  (’hunks 

-  Late  detection  of  temporally  inconsistent  situations. 

-  Slower  huiridng.  Oftmi  multiple  presentations  of  same  iiroblem. 

-  .Necessary  matching  occurring  on  im|)ossible  to  use  pioduclious. 

Recovery  -  Mere  we  have  a  s|)ectrum  of  op|)ort nnith's.  wh«>re  the  modi'l’s  tradeolfs  will  (h'liMinine 
the  mix  of  replanning  vs.  state  sharing.  This  can  chang<‘  (or  diderenl  parts  of  the  modi'l. 


•  Replanning 

-  Assumes  that  all  the  planning  decisions  should  be  remade  for  any  new  situations. 

Thus  the  stability  of  the  external  world  is  low. 

-  Results  in  shorter  elaboration  chains  as  a  rea.soning  chain  is  encoded  in  a  chunk. 

•  Sharing  State 

-  .\ssurnes  problems  will  occur. 

-  Constantly  duplicating  results,  perhaps  unnecessarily. 

-  Longer  elaborations,  each  intermediate  result  is  encoded  in  a  chunk. 

Furthermore,  the  recovery  methods  assume  that  the  organization  of  the  application  knowledge 
into  the  correct  set  of  operators  ran  be  a  problem.  The  reorganization  of  t  hat  knowledge  means 
duplicating  it  in  some  cases,  and  retjuires  the  ability  to  either  create  new  op(>rators  for  a  problem 
space,  or  to  use  the  same  operator  in  different  problem  spaces. 


3  Deliberate  splitting:  Learning  PSCM  operator  terminations 


In  our  bean  bag  tossing  example  in  Figure  2.  we  puri)osefully  ignored  the  fact  that  the  toss  from 
hand  to  hand  takes  time  in  the  external  world.  In  general  the  external  world  takes  time  to  i)ro(luce 
a  response  when  an  action  is  requested.  When  the  external  svorld's  response  time  is  included  in  an 
operator  we  call  it  the  slack  time  of  the  operator.  .An  operator  has  slack  time  when  the  operator 
requires  a  specific  result  from  the  external  world  to  continue  applying.  VVe  ('xplor(>  in  this  sectiott 
how  the  external  world's  response  time  and  the  uncertainty  of  the  duratioti  of  the  response  tinu' 
affects  PSCM  problem  .solving. 

In  addition  to  taking  titne.  theextertial  worhl  may  not  respond  as  expected.  If  a  PSC.M  operator 
needs  a  specific  result  from  the  <'Xterual  worhl  to  continue,  tlxm  the  lack  of  this  r('sult  could  k(H'p 
this  P.SCM  operator  from  completing  the  ■•<>lfective”  function  the  opmator  is  sMpi)os('(l  to  compute. 
.Vormally.  an  effective  o()erator  l«'rtninates  when  it  has  reacherl  its  desired  result.  I'o  deal  with 
the  uncertainty  of  the  external  world  i)riMlucing  tlie  desircMl  result,  we  describe  the  lu'ed  for  an 
additional  class  of  PSCM  operator  knowledge,  which  we  call  Dju  mtor  f<  niiiitdlion.  that  (hdines 
when  the  operator  has  completed.  Operator  terminatH)n  knowledge  is  a  dynamic  redelinition  of 
the  PSC.M  operator's  coitipletiou.  Somefiim's  this  t<'rmii\ation  knowh'dgc'  will  Ix'  overly  general, 
applying  at  inappropriate  times. 

When  overly  general  operator  termination  kiiowledg<‘  ap[)lies.  it  r<'mo\es  the  context  the  op¬ 
erator  provides  even  though  the  (objective  of  tin*  operator  might  still  be  achievf'd.  Inch'ed.  the 
fanatical  completion  assumirtion  wouhl  have  us  achiev<>  liiis  objective'.  To  achic'vi'  the-  obji'i  tive.  w(' 
will  describe  a  process  similar  to  the  re'cove'rv  from  uou-cont<'m|)oran('ous  chunks  in  Soar  ih'scribt'el 
in  Section  The  recovery  process  again  cenisists  of  context  n'storation  and  coul iiiiial ion  with 
previous  operators.  An  operator  i)rovid<*s  botli  a  conte'xt  for  the  internal  changes  to  the  state  and 
a  context  for  romprehen<ling  <>xternal  ehanges  to  the  stale.  This  is  the  context  that  is  lost  wh  i 
the  operator  termination  knowledge  apitlies.  In  this  se<lion  we  are  intt'rested  in  the  (ontext  lor 
comprehemling  external  changes,  because  the  lack  of  a  particular  ext«'rnal  r('sult  is  w  hat  gt'iierates 
the  operator  termination  and  r<*cognizing  the  appearance  of  the  external  result  is  the  k('y  lor  when 
to  restore  the  context. 

When  an  operator  has  been  t<'rminate<l  Ix'fore  reaching  its  objective  Ix'cause  the  external  wt)rld 
did  not  |)rovide  a  result,  we  say  it  has  Ix'en  split  by  that  termination.  .\II  the  proct'ssing  leading  up 


to  the  tenninatiou  is  the  before-split  portion.  Fhe  r<'storatioii  of  t  he  (oiitc'.xt  u[)oii  re(  ()>>,iiiziiif>,  t  lie 
result,  and  the  continuing  of  work  on  the  objective  of  the  operator,  is  the  after-split  portion.  This 
is  very  similar  to  the  architectural  splits  of  Section  2.  VVe  will  show  that  in  Soar,  splits  of  this  iyp<> 
happen  only  when  the  e.xternal  world's  results  aren't  immediately  produced,  bi'caiise  (luii'scence 
is  required  for  Soar  operator  termination  knowledge  to  be  utilized.  I’lie  definition  of  when  S<uir 
utilizes  operator  termination  knowledge  gives  us  a  more  dynamic  Soar  ojierator  that  can  adjust  to 
the  reaction  time  of  the  e.xternal  world,  splitting  only  when  recpiired  by  the  current  e.xternal  world 
interaction.  Vet  this  Soar  operator  can  still  achieve  the  original  PSCM  objective. 


3.1  Actions  take  time 

Figure  10  shows  the  problem-solving  involved  in  leari.ing  the  first  jiart  of  the  cyclic-toss  operator 
from  Figure  2  with  the  transit  time  of  the  bean  bag  included.  In  Figure  10.  tlu'  agcuit's  first  action 
request  is  to  toss  the  bean  bag  from  the  right  hand  @to  the  left  hand,  fhe  second  operator  selected 
is  to  catch  the  bean  bag  with  the  left  hand  (g).  but  this  operator  must  wait  for  ilie  beau  bag  to 
get  close  to  the  left  hand  to  apply.  As  before,  though  it  is  not  shown  in  Figtire  10.  only  when  the 
bean  bag  is  recognized  as  back  in  the  right  hand  does  the  cyclic-toss  operator  tmininate.  In  this 
e.xample.  the  movement  of  the  bean  bag  from  the  right  hand  to  the  left  takes  time.  This  time  is 
considered  to  start  when  the  agent  makes  the  action  request  0  to  throw  the  bean  bag.  and  mid 
when  the  bean  bag  is  in  the  left  hand  ©.  Since  the  movement  of  the  bean  bag  takes  tim(>.  the  agent 
has  to  do  something  while  the  bean  bag  gets  to  the  left  hand  so  that  the  catch  (left )  operator  can 
apply.  The  simplest  action  for  the  agent  to  do  is  to  wait.  This  is  shown  at  the  bottom  of  Figure  10 
as  the  e.xecution  of  the  check- progress  operator  in  the  Wait  problem  space'.  This  ik'w  problem  space 
is  created  in  response  to  the  lack  of  imtnediately  available  knowledge  to  do  anything  else  in  ('it  her 
the  juggling  or  toss  probletn  spaces  once  the  bean  bag  has  Ix'en  tos.sed.  TIh'  basis  for  formulating 
waiting  as  the  task  of  this  problem  space  is  that  an  action  reepiest  has  Ix'c'ii  made  (toss  (right  )l. 
The  check- progress  operator  in  the  Wait  problem  space  is  checking  that  progix'ss  is  Ix'ing  math' 
toward  the  goal  of  the  bean  bag  getting  to  the  left  hand.  It  is  the  prt'st'iice  of  tin'  action  re(|U('st 
and  the  recognition  of  progress  that  defines  this  impa.sse  as  slack  timt'. 

The  example  in  Figure  10  shows  that  resolving  a  lack  of  knowledge  can  I(';hI  to  ('xtt'rnal  world 
actions,  the  results  of  which  are  interpr<'t«'d  as  leading  toward  I  lit'  goal  statt'.  l  lu'^t'  i>  p("'  of 
external  world  retpiests  usually  happen  when  resolving  two  particular  typ('s  of  lack  of  know  h'dg<': 
an  inability  to  ap[)ly  an  operator  (t'xactly  the  situation  in  Figuix'  wlu'ix'  i  lu'  piM'coudit ion  o(  t  lu' 
toss  (left)  operator  is  not  met ).  and  not  knowing  what  action.s  t  lu' o|)erator  >hould  |■('(|U('st  to  leiu  h 
the  objective  (the  situation  in  liguo's  10  and  •{  for  tin'  cychetoNS  opi'iatorsl.  liotli  ha\('  to  do  with 
deciding  how  to  apjtly  the  s('lected  o|)erator  t(»  tin'  stal('.  I'SC.M  op('rators  built  in  this  way  do  not 
separate  action  recpiests  from  the  comprehension  of  tin'  results  ;t.s  both  action  r('(pi('st  and  ri'sult 
comprehension  occur  within  the  learned  I'SC.M  op('rator.  Figuix'  1 1  shows  tin'  (()mpl('t('  cyclic  toss 
operator  with  the  top  left  part  of  Figure  11  corn'sponding  to  the  (h'seription  in  Figure  10.  and  the 
right  side  corresponding  to  the  final  leariu'd  operator.  .\s  in  Figur(' s.  tin'  opc'iators  in  Figuix'  I  1 
are  shown  in  bold  with  th*'  actions  done  by  tin'  opc'rator  in  a  normal  font.  Ilowx'vi'r.  tlu'  waiting 
in  a  sub-problem  space  in  Figure  1 1  is  shown  in  italics  and  this  w'ill  Ix'  the  convention  in  this  yp(' 
of  figure  from  this  point  (hi.  In  Figure  1 1  both  toss  and  both  catch  operators  aix'  sub-op<'rators  of 
tlx'  cyclic-toss  operator.  The  waits  are  at  a  difrer«'nt  indi'iitation  l('V('l  indicating  a  snb-spaci'  wlx'ie 
[)rogress  could  be  ch('ck('d.  In  tlx'  first  case  the  agc'ut  is  waiting  for  the  prx'condit ion  <d  llx'  catili 
(left)  operator  to  be  iix't.  Figtire  1  1  shows  that  tlx'  cyclic  toss  opi'rator  will  ('ml  up  with  two  slack 
times  that  correspond  directly  to  the  tim*'  it  takes  tlx*  Ix'an  bag  to  actually  t  ravi'l  from  t  Ix'  riuhi 
hand  to  the  left  and  back  again,  ft  is  the  actual  presence  of  the  Ix'an  bag  cios<'  to  i  lx'  h'ft  hand 
that  allows  the  action  of  catching  the  bean  bag  in  the  left  hand  to  Ix'  r('<px'sl('d.  I  bis  means  i  Ix' 
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Figure  10:  Learning  tlie  throwing  of  a  bean  bag  wIkmi  toss  takes  time. 
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Figure  11:  I^oarning  when  waiting  for  resixuiscs 


operator  application  is  suspended  wliile  waiting  for  the  recpiested  action  of  tossing  to  the  left  hiiiid 
to  l)e  completed  in  the  external  world. 

In  our  example,  we  simply  checked  for  progress  tnwartls  the  (h'sirt'd  result  diiriiig  the  ^li^ck 
time.  Section  I  describes  why  you  migiit  want  to  do  other  activities  during  this  time,  and  givr-s  I  he 
methods  for  doing  so.  In  the  next  two  parts  of  this  section  we  discuss  what  happmis  if  either  I  Ik' 
agent  is  able  to  determine  that  the  result  will  never  be  achieved  or  the  rate  of  progia'ss  towards  llu' 


result  in  the  external  world  is  too  slow.  We  call  both  of  these  situations  opt  rntor  fruslntlion.  fin' 
knowledge  learned  from  operator  frustration  act  to  redefine  the  Soar  operator's  objective  allowing 
the  Soar  operator  to  be  considered  completed  prior  to  actually  arcouiplishiiig  the  operator's  P.St  '.M 
objective.  This  redefinition  is  operator  termination  knowledge.  Other  knowledge  might  also  be 
learned  when  learning  termination  knowledge  that  would  restrict  the  use  of  the  operator  iti  similar 
situations.  We  do  not  consider  that  type  of  knowledge  here. 

We  show  in  the  following  sections  that  checking  for  progress  is  a  dilficnlt  and  knowledge  rich 
process,  given  the  possibility  of  an  increasingly  hard  to  detect  difference  between  slow  [)rogress  and 
no  progress  toward  some  result  on  the  way  to  the  objective.  However  with  the  fanatical  completion 
assumption,  the  PSCM  has  a  qualitative  difference  between  slow  progress  and  no  progress.  .\o 
progress  indicates  the  objective  can  be  discarded,  and  slow  progress  iiulicates  tin'  agent  slionid  wait 
for  the  result  because  the  objective  can  be  achieved.  With  a  diminishing  difference  between  slow 
progress  and  no  progress,  mistakes  will  be  made.  Recovering  from  these  mistakes  uses  technicines 
similar  to  those  in  Section  2.3.  These  techniques  can  also  be  used  as  the  standard  methods  to 
handle  situations  where  how  to  measure  progress  is  not  known. 

3.2  Operator  frustration  over  unachievable  goals 

PSCM  models  can  refine  their  knowledge  by  interacting  with  ih<‘  (>xternal  world,  riii'si'  expi'ri- 
ences  may  indicate  that  some  aspect  of  the  original  objective's  is  cnrn'iitly  nmuhievable  and  that 
this  aspect,  and  maybe  the  entire  objective,  should  be  discardeil.  Discarding  an  obji'ct  ivi'  is  not  a 
violation  of  our  fanatical  assumption,  because  it  is  not  an  artifact  of  the  PSCM  or  SI. CM  arc  hitec¬ 
ture.  Indeed,  we  will  show  discarding  objectives  is  the  mechanism  that  lets  o|>('raior  apiilications 
continue  to  be  effective. 

.\s  an  example  of  refining  knowledge,  we  will  modify  the  cyclic  loss  ('xanijile  shown  in  Figure  10. 
Figure  12  shows  an  attempt  by  our  agent  to  apply  the  knowledge'  of  cyclic  tossing  using  a  helium 
balloon  instead  of  the  trusty  bean  bag.  .\s  shown  in  Figure  12.  this  attempt  fails  miserably,  since' 
the  balloejn  flies  into  the  air  rather  than  taking  the  standard  traje'ctory  to  the'  ot  hc'r  hand.  Ihe' 
agent  now  has  to  determine  how  to  recover  from  the'  curre'iit  situation  and  wln'tlie-r  the'  balloon 
will  ever  land  in  the  left  hand.  Re'covery  could  include'  grabbing  for  the'  balloon.  Fhe'  important 
issues  addressee!  here  are  how  the  agent  learns  that  tossing  the  balloon  up  will  not  work,  and  what 
actions  are  taken  given  that  knowledge.' 

To  determine  that  the  helium  balloon  will  ne've'r  come'  down  the'  age'iit  ne'e'ds  knowh'dge'  that 
might  have  beM>n  unavailable  prior  to  tossing  the'  balloon.  It  coidel.  howe've'r.  sugge'st  t  he  e'X|)lanal ion 
that  helium  balloons  elein't  come  back  elowei  to  the'  .same'  location.  VVilhoiil  knowle'dge'  sne  h  as  this, 
no  basis  exists  for  believing  that  the  balloon  won't  land  in  I  he'  age'iit 's  hand  some' t  inu'  in  t  he  I'nt  me'. 
Observation  by  itself  is  not  enough  to  be  sure  be'cause'  we*  are'  looking  at  a  continuous  i)roc('ss.  .\t 
some  time  we  have  to  draw  a  line  distinguishing  whe'tlie'r  the  age'nt  be'lieve's  the'  balloon  will  come' 
elejwn  or  tiejt.  Once  fhe  agent  has  ele'cide'd  that  tossing  a  hi'lium  balloon  was  a  bad  ide'a.  this 
knowledge  needs  to  be  used  to  terminate*  this  chain  of  actions  leer  ae  hie'ving  the'  geeai  of  juggling, 
fhe  catch  (left)  e>perator  ne'eeis  to  be  auginente'el  with  the'  knejwh'elge  that  it  will  be'  ine'lfe'ct ive' 
when  attempting  to  catch  a  te)sse<l  helium  ballereni.  lake'wise.  the'  e  ye  lic  te>ss  eepe'iateer  ne'e'els  to  be 
augmenteel  becau.se  it  will  alser  never  reach  its  eebje'e  i ive' e>f  a  e  ye'lie’  le)ss  of  t  he'  he'lium  balleeem.  Ifeeth 
e)f  the'se  aelelitienis  are  e)perate)r  te'rminatierti  kne)wle'elge'.  Of  e  eeurse'.  eet  he'r  kmewle'elge'  e  eeiilel  be'  h'arne'el 
that  would  alse)  refine  the  cvcfic  te)ss  e)pe'rate)r  anel  inliibil  it  wiie'ii  the'  oieje'ef  tee  (>,-  leesse-el  w,is  a 

'  The-  .iKe'Ht  eoillel.  i)f  eoiirse'.  toss  llie'  heliiiiii  tealloeui  elown  aiul  <  al<  li  it  eus  it  rise  s,  for  sim|)li<  ilv  we-  .issieieie'  oiir 
.iK<'nt  (lere's  not  leave  the  ktlowleilKe'  that  allows  it  to  i  eeiisider  tlii  lornt  ot  jiie;t;linr;. 
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Figure  12:  Frustration  when  tossing  a  helium  balloon 

helium  balloon.  We  do  not  address  the  need  to  thange  the  juggling  goal,  because  it  is  unclear  iliai 
it  is  unachievable.  There  could  be  other  jugglable  objects  available. 

Finding  that  an  objective  is  unachievable  can  happen  at  any  time  during  the  ifrobhun  sulving 
involved  in  trying  to  achieve  it.  If  the  problem  solving  is  done  completely  inlernally.  the  initial 
situation  can  be  restored.  The  e.xternal  world  cannot  always  be  restored  to  the  initial  situation.  'O 
handling  unachievable  objectives  is  mainly  an  issue  when  working  with  tin'  external  world. 

We  do  not  treat  this  form  of  frustration  further  and  liav<'  included  it  mainly  to  show  that  .1 
mechanism  must  e.xist  to  terminate  frustrate<l  operators  and  that  this  mechanism  may  need  to  b<> 
knowledge  rich.  Unfortunately,  even  if  the  knowh'dge  available  for  d(>tertnining  that  tin-  ballooti 
will  not  return  is  inadecpiate.  the  cyclic-toss  operator  in  our  example  must  lx-  terminated  as  the 
balloon  really  won't  come  down,  rermination  of  lhei»perator  without  using  domain  knowledge  is 
an  example  of  the  second  type  of  operator  frustration  frustration  oV('r  thi'  lack  of  progrc'ss. 


3.3  Operator  frustration  over  lack  of  progress 

Iti  many  cases  of  interaction  with  the  external  world,  it  is  <lillicult  to  tidl  whf'tlu'r  the  agiuit  ^hould 
continue  waiting  for  a  result  to  occur  or  airaiulon  the  [rending  oirerator.  I'his  situation  tan  ott  ur 
either  because  the  agent  has  a  lack  of  kntrwledge  about  htrw  to  measun'  progress,  or  Irecaust'  th«' 
rale  of  progress  is  slow.  In  this  typt*  of  slack  lime  iiiijrasse.  tin'  extt'rnal  world  has  bt'en  rt'tjut'sted 
to  perform  some  action,  but  tlii'  changes  (or  lack  of  changesl  in  the  t'Xlernal  world  do  not  indicatt' 
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If  The  problem  space  is  .Juggling  ami 
the  operator  is  cyclic-toss  ami 

the  toss-right  action  has  been  re(|nested  for  an  <object>  ami 
the  location  of  <object>  is  over  the  agent's  head 
then  the  cyclic-toss  operator  shonld  be  considered  coin|)lete  and 
any  objectives  established  by  cyclic-toss  should  be  nmioved 

Figure  13:  Example  of  over-general  operator  termination  knowledge 


that  progress  is  being  made  towards  the  objective  associat(‘d  with  the  action. 

We  can  modify  our  tossing  example  to  create  an  external  world  situation  of  t  his  ly[)(>  of  by  tying 
the  balloon  to  the  bean  bag.  By  changing  the  relationship  of  the  balloon’s  lift  and  drag  to  the  bean 
bag's  weight,  we  can  continuously  vary  how  long  it  will  take  for  the  bean  bag  to  get  from  one  hand 
to  the  other,  and  likewise  the  rate  of  progress  of  the  toss.  Thus,  we  can  luisiire  that  t  In'  combination 
of  balloon  and  bean  bag  will  get  to  the  left  hand,  but  that  it  will  take  an  arbii  rtirily  long  time  ( w(' 
have  a  similar  situation  when  we  are  juggling  a  helium  balloon  but  don’t  have  knowledg(>  ;ibont 
helium  balloons).  At  what  point  does  the  agent  give  up.'  .\nd  what  is  it  giving  u|)  on.’  Tossing, 
certainly,  but  what  other  components  of  the  situation  are  considered  impoitant 

There  doesn’t  seem  to  be  a  single  answer  to  when  th(>  agent  shonld  giv*'  n|>.  Tnrther.  no  basis 
exists  for  expecting  that,  when  juggling  the  balloon.  th<‘  balloon  should  lx*  considt'red  tin'  cause 
of  the  decision  to  give  up.  It  could  be  wind  or  any  number  of  otin'r  iti'ins  in  the  situation  tlmt 
are  identified  as  the  cause.  In  short,  with  no  knowledg*'  to  hel[).  tin'  agent  can’t  d('t('rmine  how  to 
assign  credit  or  blame.  In  our  case,  if  the  helium  balloon  cannot  In'  assigned  blame  for  the  problem. 
then  the  termination  knowledge  might  be  applicable  as  soon  as  any  obi<'ct  is  tossed. 

The  agent  could  tak(' t  In'  radical  approach  of  t<'rminating  t  he  op<'raior  as  soon  as  it  linds  t  hat  it 
cannot  measure  any  [)rogress.  This  approach  has  two  probh'ins  that  can  In'  illustrat('d  with  juggling 
the  balloon  beau  bag  combination.  The  first  is  that  wln'ii  tin'  ballooti  In'an  bag  combination  comes 
down  in  the  left  hand  the  agent  should  be  able  to  o'cognizc'  that  this  eva'iit  occnrreu  ln'cans('  tin' 
balloon  bean  bag  coud)inatiou  was  thrown  in  tin'  |)ast.  and  that  this  throw  was  |>art  of  a  cyclic 
toss  so  that  the  balloon  bean  bag  combination  can  In'  tossed  back  if  it  is  still  appropriate'.  This  is 
a  direct  inference  from  the  fanatical  completion  assumption.  Tin'  fact  that  tin'  b.dloon  did  conn' 
down  means  that  the  operator  did  not  need  to  be  te'rminated.  atnl  tints  the  h'arned  ti'iiiiinat ioti 
knowledge  is  over-general  and  unfijrtnnately  will  be  applicable'  in  eethe-r  sit mit ieuis  wln'ie'  waiting 
ce)uld  possibly  be  more'  appreipriate.  .\n  e-xample  e»f  e>ve'r->g('ne'r:il  te-rtninat ieen  knetwh'elge'  fetr  the' 
cyclic  toss  eeperator  is  given  in  Figure'  13  whe-re'.  any  time'  t  he>  etbje'et  be-ing  teisse'el  gete's  eeve'r  the- 
agent’s  heael  the  cyclic  te>ss  e)pe'rate)r  will  be-  ceensieh'n'el  eeemph'te'el.  In  tin-  m-xt  se'ctieni  we-  will 
briefly  touch  ean  the  proble'in  of  re'cetvering  freim  an  eeve'r -ge'tie'ial  oix'nilor  te'rminaf ietn  te)  ri'ae  h  the' 
original  ejbjective.  Beceivery  he're  is  similar  te)  that  in  Se'etieni  '2.3  .tinl  it  me'aiis  se'tting  up  the- 
conelitions  so  that  work  e)n  the'  eudgitial  e)bje'etive'  ean  ee)nl iiieie'.  Tin'  se-eetud  pie)ble'm  is  a  lorni  of 
the  masking  preibh'iii  ['Tamin' ane)  l{e)senbloe)m.  11)0.3).  Sinee'  tln'age'iit  slntidel  eise-  t  In' e>xpe'riene  e's 
in  the  extc'rnal  we)rlel  te)  builel  up  its  kne)wle'elge'  base*.  se)me'time'  iu  the’  future'  the-  age-nt  woiihl  want 
tf)  create  me)re' speeific  kne)wleelge'  that  utilize's  t  In' e’X|)aiiele'el  kne)wle'elge’  base’.  Iloweve'r.  the' original 
terminatie)n  knowh'elge  will  mask  the'  ne'w  kneewh'elge  be'caiise'  an  impasse'  will  met  oee  ur  .illeewing 
the'  new  kne)wle'elge  te)  be'  utilizeel.  VV’e'  will  ne)t  aeleiress  the-  masking  pre)ble'm  fnitln’i  in  this  |)a()e’i. 


3.4  Recovering  from  an  over-general  operator  termination 


Terminating  the  PSCM  operator  means  that  the  context  tlm  operator  pfovi(l('s  is  not  available  for 
interpreting  what  is  happening  in  the  external  world.  .\s  an  exami)le.  suppose  a  high  toss  of  a  bean 
bag  comes  down  into  the  left  hand  some  time  after  the  catch  (left )  o|>eratt)r  ;ind  cyclic-toss  operator 
have  been  terminated  by  the  knowledge  in  [-'igiire  l.'f.  Ilow  does  the  agent  know  to  continue  with 
the  next  toss?  To  continue,  the  agent  has  to  understand  tlial  this  change  in  the  ('xternal  world  is  a 
result  of  the  toss  (right)  action  retpiest  and  is  part  of  a  cyclic-toss.  Thus,  to  understand  what  the 
bean  bag  coming  into  the  hand  means,  the  PSCM  context  has  to  be  rc'slorc'd;  then  the  proc<'ssiug 
can  continue  if  it  is  still  approitriate. 

In  Section  2.. I  vve  showed  two  different  methods  for  restoring  the  problem  solving  context  after 
a  non-contemporaneous  chunk  didn't  apitly.  that  let  us  recover  from  the  disrui)lion  caused  bv  tlu' 
non-contemporaneous  chunk.  Once  the  contc'xt  was  restored,  theti  the  problem  solving  continued 
naturally,  and  the  objective  of  the  operator  was  achieved.  The  (irst  method  ic’sioia'd  the  context 
via  re-planning  from  the  original  objectives  and  tlu*  new  situation.  The  second  method  continuallv 
saved  the  state  of  the  planning  activity,  so  that  the  probhun  solving  cont('xt  was  imnu'diaielv 
available  when  the  disruption  occurred.  Vari.ations  of  hot h  tlx'se  imuhods  can  b(>  used  in  recf)vering 
from  the  over-general  operator  frustration  knowledge,  providing  us  again  with  a  continuum  of 
solutions.  Re-planning  is  exactly  the  same  as  in  Section  2. .'5  and  gemuuiti's  a  new  cont(>xt  that 
is  like  the  old  one,  different  only  in  that  new  symbols  may  b(>  geiieriited  for  the  same  objects. 
However,  the  method  of  sharing  state  in  Section  2..J  doesn't  work  directly  b(>c;tuse  the  termitial  ion 
knowledge  removes  all  t  races  of  t  he  tcuuninated  operator.  Instead,  befoiu' t  he  op('r;>t()r  is  t  ('rminated. 
appropriate  portions  of  t he  context  can  lx*  m<‘moriz<'d.  Fhe  memori/,('d  contc'xt  can  then  bc'  usc'd 
to  recreate  the  context  at  the  coirect  time'.  This  memorization  and  ri'-creat ion  method  is  sindlar 
to  the  sl\ared  state  of  S<'ct ion  2.d  because'  tlie  probh’in  solving  coutiniK's  from  wh('r<‘  it  was  stoppe'd 
without  re-doing  any  of  tin'  planning. 


Restoring  the  context  through  planning 

.Mitchell's  robot  systi'm  [.Mitchell.  1!)9()]  archile'cl iirtdly  |)lans  until  an  action  is  known  to  lu' 
on  the  path  to  achieve'  the'  geeal.  then  it  always  te'indnate'  the'  erpe'iator.  and  re'-plaiis  leer  the'  ne'xt 
action,  figure  S  slurwed  Mitehe'll  s  syste'in  le'arning  the'  e'velic  te>ss.  igne)ring  the’  e'xte'iiial  weerlel 
re’speense’  time’.  Figure  11  slieews  heiw  a  PS('.M  imple'me’nlatie)n  of  Mite  he’ll's  me’tluxl  would  h'aru  the’ 
eye  lic-teess  erpe’rator  whe’ll  e'Xte’rnal  Weerlel  re'speeiise'  time'  is  eeuisiele’fe'el ,  .\s  in  the’  preeble’llt  seiKiiig 

trace’  where  the  agent  waile’el.  Figure-  1  1.  the  h’l't  hanel  siele-  of  f  igure’  1  I  sheews  the-  preible-m  seelving 
trace  while  le’arning  anel  the’  right  haiiel  side’  sinews  it  afte-r  le’arning.  lu  Figure-  1  I  the-  e  yelie-teess 
ejpe-ratejr  is  te-rminate-el  as  seeeen  as  a  re-al  aetieen  re’e|ue’st  is  maele-,  1  his  pelts  wailing  lor  a  le-speuise- 
eeutsiele  eef  any  eef  the-  ope-r;itors.  as  sheewn  in  f  igure  I  1  by  tin-  lae  k  eef  inele-ntal  ieui. 

Re-creating  a  previous  context 

I  in-  eeeule-xt  that  ne-e-els  lee  be-  r'-store-d  bv  re’eeeve-r\’  ineluele-s  the-  ob  je-ct  i\ e-s  of  tin-  ope-rateu  li.e-.. 
the-  geeal  that  de-se'ribe-s  what  tin-  eepe-rateer  is  living  lee  aellie-ve-i  ,iuel.  |eeessibly.  eiiln-r  slate-  iiileerma- 
lieen.  I  he-  ae  tiial  eepe-ialeer  eloe-sn  t  lie-e-il  lee  be-  |e-sleer<-il  il  [ereeee-ssillg  leer  the-  lebje-e  live-  e  au  eeinlinue-. 
Re-steering  the-  e'eenle’Xl  wln-ll  a  re-sull  is  eebse-rve-d  re-ipiir<-s  seeiin-  lin-l  Ineeleeleeuv  leer  I  e-e  e  eia^U  i  Z  i  U  U,  W  In-u 
an  e-xte-rnal  e  hange-  sinelllel  be-  e  eensii|e-re-e|  a  re-sul(.  l  ids  llie-t  Inedeeleigv  e  an  be-  kneevv  le-elge-  le-an  anel 
assiimptieen  rieh.  assuming  tin'  first  eebse-rvable-  ehange-  in  tin’  e-xie-rual  vveeild  is  tin-  n-suli  eef  tin- 
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Figure  Id:  Learning  when  always  re-planning  and  actions  lake  liiiK' 

requested  action.  The  methodology  can  also  use  various  knowledge  sonret's.  such  as  prt'vions  ('xpe- 
rience  in  this  domain,  or  an  internal  model.  Tlie  key  is  not  the  restoring  of  the  context,  for  that  is 
available  at  the  time  the  termination  knowledge  is  learned,  but  rather  recognizing  trln  n  t  h('  contcvxt 
should  be  restored,  as  this  situation  is  not  available  when  the  tr'rminalion  knowh'dgr'  is  generati'd. 

(.'reating  the  e.xpected  situation  and  then  using  this  created  situation  as  tin-  soiirc<'  of  knowl¬ 
edge  of  when  the  restoration  should  occur,  is  one  way  to  geiK'iatr'  the  r«'storation  knowledgi'.  In 
our  example,  the  expected  res|)onse  is  the  bean  bag  (or  balloon)  in  the  h'ft  hand.  \\e  want  tin' 
recognition  of  this  response  to  lead  to  continuing  the  cyclic  toss.  To  continnc'  tin'  cyclic  toss  we 
need  to  install  the  toss-twice  objectivt'  and  mark  lh<'  obje<  liv<'  to  indica,t<'  the  object  has  comph'ti'd 
one  toss.  Thus,  the  restoration  knowhulge  might  be  what  is  shown  in  I'igurc'  IT).  I'In'  knowledg*'  in 
Figure  15  restores  the  context  that  was  available  when  the  opc'rator  was  lenninal<'d.  Once  r<'siored. 
the  catch  (left)  operator  can  apply,  completing  the  in'Xl  step  low.uds  tin'  original  op('ratoi  's  (dii<'c 
tive  (do  cyclic-toss).  'The  objectiv<'  can  Ix'  r('moved.  if  nec<'ssary.  by  the  satin'  proct'ss  dest  ribt'd  in 


If  The  problem  space  is  Juggling  and 

the  toss-right  action  has  been  requested  for  an  object  and 
the  status  of  the  object  is  moving 
the  location  of  object  is  close-to  <han<l> 

<hand>  is  the  left  hand 

then  the  cyclic-toss  operator  should  be  propcised  witli 

the  toss-twice  objective  insfalle<l  as  a  sub-task  of  juggling 

Figure  15:  Example  of  over-general  resloialiou  knowledge 
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Figtire  Hi:  Leartiing  when  saving  i)laiining  state 

Section  2.5. 

.Note  that,  just  like  the  termination  knowledge  in  Figure  15.  the  restoration  knowhulge  show  in 
Figure  15  is  overly  general.  .Some  time  in  the  future,  when  the  agent  is  juggling  and  the  jugghul 
object  i.s  close  to  the  left  hand,  the  agent  will  suddenly  have*  thi'  goal  of  doing  the  last  half  of  a 
cyclic-toss!  The  problem  is  that  the  restoration  knowledge  is  not  linked  to  (he  initial  oju-raloror  to 
the  specific  type  of  juggling  goal  that  led  to  the  cyclic-loss  operator  being  proposed.  This  linkage 
can  (‘ither  be  explicit,  conveyed  through  some  constant  shared  by  this  spc'cilic  juggling  goal  and  the 
restf)ration  knowledge,  or  by  linking  to  the  previous  slej)  in  some  ntaniK’r.  or  through  some  more 
complex  mechanism  like  an  intention.  We  will  assume  that  tin'  ii'storatiou  knowledge  (  an  be  math' 
specific  enough. 
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Figure  l(i  shows  a  problem  solving  trace  lor  doing  the  cyclic  loss  both  before  and  after  learning 
wlien  the  state  of  tlie  plan  is  saved.  Its  format  resembles  the  other  traces  except  that  the  start 
of  slack  time  is  explicitly  represented  because  the  slack  time  starts  as  part  of  the  o|)erator  and  is 
continued  after  the  operator  is  terminated. 

The  trace  in  Figure  16  begins  like  the  trace  in  Figure  11  until  the  slack  time  occurs.  When 
slack  time  within  an  operator  is  initially  detected,  rather  than  just  waiting,  three  operators  apply. 
These  operators  generate  the  knowledge  to  both  split  the  operators  above  and  recognize  wlnui  the 
proper  external  world  events  have  occurred  so  that  the  context  can  be  restored,  rite  visualize- 
result  operator  in  effect  does  a  one-step  lookahead  to  determine  the  action's  expia  ted  result.  In 
this  scenario,  the  toss-right  is  the  only  action  request  that  has  been  madi'.  so  visualize- result 
changes  the  current  situation  to  one  showing  the  bean  bag  close  to  the  left  hand.  UuHd-recognition 
then  generates  restoration  knowledge  such  as  that  shown  in  Figure  la  for  the  cyclic-toss  operator 
using  the  visualized  result  and  portions  of  the  current  context.*  Iliiild-terminalion  creates  the 
termination  knowledge  for  the  cyclic-toss  oper.ator.  like  that  shown  in  Figure  l;{.  but  not  based 
upon  any  domain  knowledge.  The  termination  knowledge  for  the  catch  (left)  oj)erator  is  based 
solely  upon  the  existence  of  slack  tinte.  Slack  time  in  this  case  is  recognized  by  both  the  lack  of 
immediately  available  knowledge  and  the  fact  that  an  object  was  reciuested  to  be  tossed  from  the 
right  hand.  The  cyclic-toss  operator's  termination  knowledge  is  based  upon  the  same  information  as 
the  catch  (left)  termination  knowledge,  and  the  non-existence  of  alternative  operators  to  implement 
the  cyclic-toss.  This  termination  knowledge  applies,  terminating  th(>  operators  ii\  Figure  Mi.  I;:;f 
doesn't  end  the  slack  time  (the  external  world  has  to  respond  for  t  h(>  slack  lime  to  be  over),  fhus. 
the  agent  now  waits  for  the  bean  bag  to  come  close  to  the  left  hand  outside  of  the  oixMi.lor.  I'hese 
three  operators  have  split  both  the  cyclic-toss  and  catch  (left )  o|)<>rittors.  whih'  building  i  In’  reco\<'ry 
mechanism  that  keys  off  the  expected  result  of  tin*  bean  l)ag  being  close'  to  tin'  h'fl  h;ind.  I'his  is 
the  basic  mechanism  of  saving  state. 

We  continue  the  processing  in  Figure  16  when  the  bean  bag  is  close  to  the  h'ft  hand  because 
the  recognition  knowledge  re-establishes  the  removed  context.  TIk'  context  is  ('xtictly  the  same  as 
before  the  split,  enabling  the  agent  to  select  the  cyclic-toss  opertitoi’  agtiin.  ;tnd  the  (atch  (left) 
operator  under  it.  .After  catching  with  the  left  hand  and  noting  the  inl('rme(lial<'  result,  the  agent 
still  has  the  objective  to  toss  the  bean  bag  back  to  the  right  hand,  and  the  loss  oix'itilor  is  used 
to  begin  achieving  this  goal.  Once  tossed,  we  again  have  slack  time,  this  lime  occurring  within 
the  catch  (right)  operator.  This  catch  o()erator  and  the  cyclic-toss  opi'iaior  aix'  again  split  in  a 
similar  tnanner.  The  previous  chunks  did  not  apply  because  they  wc're  depemh'ul  iii)on  i  hi'  left 
hand  catching,  not  the  right.  Finally,  when  th«*  l)ean  bag  is  close  to  the  right  hand,  it  is  caught 
and  the  cyclic-toss  is  completed. 

The  right  hand  side  of  Figure  16  is  much  simpler.  It  shows  I  he  cyclic-toss  operator  being  si'lected 
and  applied  three  times  in  achieving  the  cyclic-toss's  objective.  Once  the  loss  (rom  the  right  hand 
is  rerpiested  the  cyclic-loss  operator  is  terminated  by  the  previously  learned  li'rmiuation  knowh'dgi'. 
fids  knowledge  also  removes  the  objectives  from  considi'ration.  When  the  bean  bag  gel  s  close  to  I  hi' 
left  hand,  the  restoration  knowledge  applies.  r«'-crea.iing  the  context,  and  the  cyclic-loss  operator 
is  selected  and  applied,  this  time  catching  the  bean  bag.  noting  the  intermediate  state  and  tossing 
it  back  to  the  right  hand.  This  toss  also  takes  lime,  so  this  instance  of  llie  cyclic-loss  operator  is 
also  terminated.  Finally  when  the  bean  bag  a|>proaches  the  right  hand,  the  last  instance  of  the 
cyclii  -toss  operator  is  selected,  catching  the  bean  bag.  and  achieving  t  hi'  cyclic-toss's  objective. 

^(’ri-atiiiK  this  kiiowlc(li>c  ri'(|iiircs  data  chiinkiiig  [Ni  wcll,  I'MMl]  l he  < oiiti  xt .  Ilowi  vi  r.  in  lliis  iiisi.iinc  we  have  a 
i>eneralor  for  the  context  in  the  original  conlext-laiiUlitig  product ion.s. 
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Figure  17:  Number  of  operators  with  splitting  at  I’SCM  slack  time 

Continuing  the  PSCM  operator 

Continuing  the  work  on  the  objective  of  the  PSCM  operator  involves  a|)plying  otlier  oi)erators.  An 
important  issue,  then,  is  whether  the  correct  operators  will  he  available.  Our  cfuitcuitiou  is  ilnw  are 
available  because  the  applicable  operators  (after  restoring  the  state  if  necessary)  are  exactly  those 
that  were  originally  available.  In  the  left  hand  side  of  Figure  Hi.  the  lirst  split  occurs  after  the 
bean  bag  is  tossed.  .A.  new  proposal  for  the  cyclic-toss  operator  was  created  by  tin*  l)uild-recognition 
operator  during  the  slack  time.  This  proposal  makes  the  cyclic-loss  operator  appropriately  available 
at  the  end  of  the  slack  time.  The  catch  (left)  operator  is  available  because  the  context  provided  by 
the  recognition  knowledge  is  similar  to  the  original  context 


3.5  Dynamic  operators 

VVe  have  shown  that  when  interacting  with  an  external  world  that  has  uncertainty  in  the  results 
it  provides  to  actions,  then  operator  termination  knowledge  tmist  be  learttable.  However,  disfio- 
guishing  between  tlie  situations  where  operator  termination  knowledge  should  Ix'  h'arued  and  thos*' 
where  it  should  not  is  difficult.  .Also,  the  operator  termination  knowledge  may  be  oviu  -geueral.  ap¬ 
plying  in  situations  that  are  similar  to  the  learning  situation  but  that  don't  nMiuire  t  1h>  termination 
knowledge.  To  recover  from  the  application  of  operator  termination  knowledge  and  continue  the 
problem  solving  rerpiires  that  the  context  that  existed  before  the  termination  be  restored.  The  use 
of  operator  termination  knowledge  separates  the  knowledge  in  an  operator  into  two  components;  an 
action-trquest  component,  that  determines  what  action  shouhl  be  doin',  and  a  comprehend-ri'snlt 
component,  that  completes  the  problem  .solving  a.s.sociated  with  tin'  original  operator.  In  this  sec¬ 
tion  we  show  that  if  operators  are  always  split  when  an  external  world  action  recpiest  is  made,  then 
the  number  of  operators  executed  to  achieve  some  result  can  grow  t  r('nn'ndonsly.  .\lt  hough  growth 
can  be  moderated  by  splitting  in  a  less  pedantic  manner,  the  reason  for  splitting  remains:  progress 
in  the  external  world  is  f(X)  slow.  In  short  the  external  world's  response  time  ()('lines  the  grannlaritv 
and  number  of  PSCM  operators  that  a  model  uses  to  do  a  task. 

Figure  17  shows  the  number  of  operators  used  to  achh've  two  tasks  wheif'  eit  her  I  he  operators  are 
always  split  after  an  action  re(iuest  or  never  split.  The  lirst  task  is  the  familiar  cyclic  toss  example 
and  the  second  is  a  more  complex  task  of  a  robot  pushing  a  box.  I'he  model  used  for  each  task  sent 
action  rerpiests  to  the  external  world  at  each  st»'p  of  its  planning  and  used  tin'  results  to  v('rify  that 
the  plan  was  working  according  to  its  expectations.  I'he  data  in  the  first  and  sf'cond  (olnmn  (onu's 
from  a  Soar  system  that  always  splits  the  operator  whenever  an  action  reipiest  is  made,  and  then 
restores  the  context  when  the  desireil  result  beiomes  available,  so  that  tlu'  |)roc<'ssing  toward  tlie 
operator's  objective  can  continue.  ’  I’he  data  in  the  third  c«)lnmn  comes  from  a  similar  Soar  system 
that  never  splits  the  ojierator  when  an  action  reipn'st  is  imnh'.  but  insti'ad  waits  until  the  desired 

'  I’lie  robot  “Always  split  with  rc-planiiinit"  is  a  liand  siiuiilatioii  ami  (lie  ".Mivavs  .-I'lil  saved  -l.ilc"  iiii/>)eim  ii(alii)ii 
.actually  saved  the  planniii);  stale  rather  than  Kettiiii;  rid  of  it  at  teriiiiiiatioii  and  later  resloriii);  it  . 


response  is  found  in  the  situation,  allowing  the  operator  to  continue.  Figure  17  sliows  iliat  always 
splitting  means  many  more  operators  in  both  recovery  scenarios.  Fhese  e.xtra  operators  can  limit 
how  effective  the  agent  can  become  in  doing  some  task,  'fhis  limitation  shows  up  if  oixuators  ar<' 
being  assigned  some  real  time,  like  50  msec  [Newell.  lOOO],  so  that  a  model  can  be  coirelated  with 
data  from  people.  The  e.xtra  operators  may  make  the  model  overpredict  tin'  data.  Fhe  limitation 
also  rears  its  head  when  doing  planning,  for  planning  is  a  NP-liard  task  [Chapman.  1!)S7]  and  more 
operators  reduces  the  effectiveness  of  any  planning  activity.  I'he  number  of  ofrerators  becomes  even 
higher  as  the  granularity  of  interacting  with  the  external  world  becomes  finer.  I'o  ease  this  growth 
effect,  we  notice  that  we  only  want  to  split  operators  when  no  i)rogress  is  being  made  during  slack 
time,  not  at  every  action  request.  The  number  of  oirerators  used  by  a  modcd  that  us<'s  this  splitting 
philosophy  to  do  a  task  woukl  be  somewhere  between  the  never-split  and  one  of  the  always-split 
cases.  Since  splitting  in  this  model  is  dependent  upon  how  (piickly  t  he  ('xt('rual  world  r<'sponds.  t  he 
number  of  operators  is  ultimately  dependent  on  the  respon.se  time  characteristics  of  the  r-xternai 
world. 

The  number  of  operators  to  do  a  task  would  seem  to  only  grow  via  splitting.  However,  one  of 
the  interesting  properties  of  operator  terminatit)n  knowledge  is  that  PSC.M  operators  split  by  such 
knowledge  can  recombine  to  become  a  single  o()eralor  again,  if  the  PSC.M  is  defined  as  waiting 
for  all  immediate  knowledge  to  apply  before  selecting  a  new  operator.'  IC'combining  happens  if 
the  external  world  responds  immediately,  so  no  slack  time  exists  in  t h('  prc'viously  si)lit  PSC.M 
operator.  Thus  the  conditions  may  be  right  for  operator  termination,  but  with  tin'  I'xternal  world's 
response  available  the  rest  of  the  knowledge  as.sociated  with  the  PSC.M  opmator  (an  still  aftply.  If 
the  operator  is  allowed  to  continue,  delaying  the  use  of  operator  t('rminiit  ion  knowledge',  i  lu'ii  this 
e.xecution  of  the  operator  can  achieve  its  ol)j('ctive  in  the  normal  iminner.  This  |■('combining  of  an 
operator,  after  it  was  split,  again  shows  how  the  response  of  the  exK'rmd  world  delines  the'  timouni 
of  work  an  ojterator  can  achieve. 

.\s  a  final  observation,  when  working  completely  with  intt'rnally  generate'd  n'sulis  (e.g.  simu¬ 
lating  a  sequence  of  events  as  in  planning),  split  operators  rt'combine.  This  happens  Ix'cau.sf'  an 
internal  model  of  the  external  world  |)rovides  the  ('xpected  respons('  to  an  action  re(|nesl  imim'di- 
ately.  This  recombining  makes  internal  proc('ssing.  like  planning,  simph'r  Ix'caiisi'  the  mimix'r  of 
operators  is  ke]>t  low. 


4  Splitting  with  Multiple  Objectives 


In  Section  •'{  we  were  concerned  oidy  with  checking  that  progress  was  being  made  during  slack  time. 
However,  if  we  have  multiple  tasks,  waiting  during  slack  linn'  could  Ix'  consith'O'd  iix'llic T'lit  ns(' 
of  the  cognitive  resources  available.  I'he  PSCM  has  a  s<'<|uenlial  opc'iator  hot t h'tx'ck.  that  is.  it 
executes  just  one  operator  at  a  time.  When  waiting,  tlx*  PSCM  is  ol)s('rving  progo'ss  Ix'ing  mad('  in 
one  of  its  tasks.  Ifather  than  just  observe  progress.  j|  might  Ix'  able  to  nmki'  progrc'ss  on  a  differt'tit 
task  and  return  to  tlx*  original  task  sometinx'  later.  I'o  achieve'  maximum  us<'  of  i  Ix'  cognitive' 
re'source.  we  would  like  to  have  the'  PSC.M  operate*  on  one*  of  its  ot  Ix'f  tasks  wlx'ii  the'  work  on  tlx* 
curre'iitly  e'xeciiting  task  is  suspe'ixle'd  (waiting  lor  an  re'sult  to  lx*  obse'ive'd).  This  se'ciion  eh'scrilx's 

'Soar  i  iirrcnl.lv  works  this  wav.  IxX  I  lie  (Iclinltioii  ol  llic  I’St 'M  is  nne  lcar  alioiil  when  avail. ililc  o(>cralois  .ere 
coiisidcrcel  for  sclce  lioii.  A  I’.Se’M  operator  •onlel  liiipleineni  several  fniK  lions,  e.i<  li  delineil  hy  ilillereni  operator 
termination  knowled)>e.  Over  time,  ^is  1  he  ope'raterr  ap|)lies.  dilferent  termination  knowledi;e  i  an  liei  ome  .applie.ehh' 
possiblv  starting  the  seleetion  process  for  a  new  o|)eralor.  I’he  PSt’M  is  unclear  on  whether  the  lirsi  inslaiuc  ol 
applicable  termination  knowledge  starts  the  selei  tiejn  process,  or  if  some'  othe'r  e  eindition  ol  ihi-  l’S(  M  .ir<  liile  e  l lire- 
is  used  to  determine  that  some  of  this  knowledge  e  an  l>e  igliori'il.  Soar  uses  ipiiescenci-  to  ililermine  the  lermin.ition 
knowledge  that  ilehnes  the  termination  i onelilions  ol  an  instance  of  .in  o)>erator 


Objectives  Tasks  Operators 

Figure  18:  Multiple  01)je(  tives  aiul  Multiple  Tasks 

what  keeps  this  transfer  of  control  from  happening  in  the  PSCM.  then  discusses  two  solutions  and 
how  models  built  using  those  solutions  are  related.  By  transferring  the  control  between  inulti])|e 
tasks  we  are  able  to  use  the  available  cognitive  resources  efficiently. 

A  task  is  an  abstract  description  of  a  process  to  achieve  an  objective.  In  I’SCM  terms,  a  task 
can  be  represented  by  either  a  large  operator  that  may  include  many  steps,  or  multiple  smaller 
operators  each  doing  only  a  few  steps.  When  working  with  multiple  objectives,  either  one  has  a 
single  task  that  can  achieve  all  the  objectives,  or  multiple  tasks  that  one  hoi)es  can  achi('V('  all  the 
objectives  together.  Figure  18  shows  a  PSCM  model  with  multiple  objectives,  the  lirsi  two  of  which 
are  associated  with  a  single  task  that  has  multiple  operators  to  implement  it.  ’  The  third  objective 
is  resolved  in  a  task  that  has  only  a  single  operator,  while  the  fourth  obj<>ctive  in  Figure  IS  has  two 
operators. 

When  working  with  multiple  objectives  and  multiple  tasks.  (>.\ternal  (>vents  that  occur  while 
processing  any  of  these  tasks  can  either  indicate  that  the  desiral)ility  of  working  on  one  of  the  tasks 
has  changed,  that  the  application  of  one  of  the  operators  cati  proceed,  or  that  the  achievement 
of  a  objective  has  occurred.  .-\s  an  e.xample.  the  externai  world  could  change  (my  tmiiiinal  might 
burst  into  flames)  so  that  an  important  task  (saving  my  skin)  should  intc'rrupt  a  less  important 
task  (typing  these  words).  In  PSCM  terms,  the  changitig  of  the  ('Xternal  world  would  make  the 
operators  for  fleeing,  getting  a  fire  extinguisher,  etc.  more  salient.  We  want  these  more  salient 
operators  to  be  selected  and  applied,  ignoring  what  we  are  currently  doing.  In  general,  we  would 
like  the  processing  of  multiple  tasks  at  the  PSCM  level  to  occur  as  just  a  se<]uence  of  optuator 
selections  followed  by  their  a|)plication.  always  working  on  the  most  salient  task. 

Unfortunately,  it  is  unclear  how  such  a  se<|uence  of  PSCM  operators  could  taki'  advantage  of 
the  slack  time  of  the  operators.  Utilizing  slack  time  within  an  ojx'iator  im'ans  that  om>  of  the 
following  must  occur: 

•  Split  the  PSCM  operators  with  slack  time  into  a  se«|uence  of  smaller  a(t  ion- ret]  nest .  comi)r('h<'nd- 
result  operator  pairs. This  split  uses  the  operator  termination  mechanism  descrilx'd  in  Sec¬ 
tions  8.2  and  8.8  and  the  recovery  mechanism  in  Section  8.1  to  r('nu)V('  tin'  sbn  k  lime  from 
the  operator  but  allow  the  task  to  continue  processing  in  the  future'. 

This  o()tion  removes  the  slack  time  Irfem  the  ope'ralors.  I'lius  w('  inle'rh'ave'  small  ope'iators. 
and  a  lack  of  something  to  do  is  the  only  reason  to  wail. 

•  C<)rtibine  operatf)rs  from  multiph’  obje'ctivi’s  into  large'r  opi'iators.  Siiue'  I  In'  PSCM  has  no 

'This  is  a  fiiodihralion  of  Fij^iirc  I  in  (t 'ovrinarii,  IWtjj. 

'  If  the  external  world  is  very  e  losi-  to  the  internal  model  lln-ii  perhaps  tlie  result  does  not  need  to  In-  i ompri  liended 
.ind  just  a  se<|iience  of  action-re<|nests  is  ni'cdi'd.  riiroii(;hoiit  this  paper,  tlioiit'li.  we  will  reler  to  lliese  op<  ralors  .is 
pairs. 
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stMiuentiality  r('(|iiiroiiHMits  witliiii  an  oix'iator  ai)i>li<atioii.  only  tlin  s<-<|ih'iiI iallty  a|)|)ai<Mii  in 
the  (lata  dependencies  will  n'strict  mnlliple  objectives  from  Ix’infi,  pursued  in  paral|('l.  I  lli^ 
creates  situations  like  at  the  top  of  Figure  IN  where  the  first  two  objeclivf's  ar<'  proc('ss(‘d  by 
the  same  task. 

This  option  overlaps  all  the  available  processing  so  that  it  occurs  within  a  single-  operator. 
Thus,  any  waiting  that  exists  after  combining,  is  due  simply  to  lack  of  something  to  do. 

•  Break  the  sequential  .selection  and  application  of  operators  allowing  mullii)le  oix-rators  to  lx- 
applied  simultaneously  in  the  same  problem  s|)ace.  Ro.senbloom  has  |)ropos('d  changing  Soar 
to  allow  this  [Rosenbloom.  l!)f):l]. 

This  option  also  overlaps  all  the  available  prori-ssing.  so  that  it  occurs  simultaiK'oiisly.  How¬ 
ever.  it  provides  some  architectural  assistance  to  the  handling  of  the  simult aix'ous  tasks. 


Investigating  the  implications  of  supporting  multiple  simultaiu'ous  operators  is  Ix-youd  t  he  scope 
of  this  paper.  The  next  two  parts  of  this  section  investigate-  the  first  two  methods  and  how  the>  ar(- 
related.  VVe  look  at  splitting  operators  at  slack  time  and  tlx-n  int<-rl('aving  first.  I)ecaus(-  tlx-  splitting 
portion  should  be  familiar  by  now.  The  interleaving  is  fairly  simple  and  all  tlx-  iix-chaiiisms  for 
interleaving  are  in  the  Soar  and  PSC.M  architectures.  However,  interactions  can  (-xist  lx-tw(-(-n  t  Ix- 
two  tasks.  .As  an  example,  given  the  two  olyjectives  of  painting  tlx-  ladder  and  tlx-  ceiliixg.  tlx-  ag(-ni 
should  paint  the  ceiling  first.  This  is  a  real  problem  when  working  with  si>lit  o|x'ralors  becaiist- 
during  a  split  we  have  removed  the  information  that  other  operators  could  us(>  to  const  rain  tlx-ir 
behavior.  .After  interleaving  we  look  at  two  meilnxls  (*f  comitining  operators,  one  that  r('(piir(-s 
the  knowledge  associated  with  a  strong  nxxlel  of  the  int(>ract ions  lx-iw(-(-n  tasks,  atid  one  that 
doesn't.  We  show  that  the  knowh'dge-U-an  system,  breaks  both  tlx-  PSC.M  conc(-pt  of  an  oix-riitor 
implementing  a  function  from  one  state  to  another  and  the  faitatical  completion  assumption.  I'lius 
we  discard  it.  The  knowledge-rich  strong  nxxlel  system  conx-s  from  [Covrigaru.  I!)'»2].  atxl  wi- 
review  it  as  a  method  of  planning  actions  and  paralh-li/ing  iixlepc-txh-tii  interactions  with  tlx- 
external  world. 


4.1  A  second  task:  Robot  pushing  a  box 

We  begin  by  modifying  our  tossing  (-xample  so  that  wc-  have-  mulliph-  oix-r;itors  a\ailal)|(-.  In 
addition  to  tossing  a  bean  bag  from  haixl  to  hand.  w('  mak('  our  agc-nl  a  rol)ot  .nxl  give  it  a  second 
objective  of  pushing  a  box  from  room  to  nx»m.  .\n  i-xamph-  prol)l(-m  in  t  h<-  robot  dotnain  is  show  ti 
at  the  top  (jf  Figure  1!).  .At  the  bottom  of  Figure  H).  tlx-  lirsl  >t(-ps  toward  solving  tlx-  piobh-m 
are  shown.  The  Push-Box- I’liruf  Door)  operator  has  lx-<-n  M-|(-ct(-d  (.\)  in  tlx-  mov<-  robot  probh-m 
space  as  the  first  operator  try  for  this  task.  I  his  operator  canix)i  -ii'ply  b(-caus(-  of  an  unia-solvc-d 
precondition.  .As  before,  the  I’SCM  casts  t  lx- lack  of  kix»wi(-dg(- of  how  to  ri-soKa-  this  jirec  ondil  ion  .is 
a  task  to  be  solved  in  a  itrobh-m  spac('.  Tlx*  task  is  formnlat(-d  to  havi-  an  initial  state  ( t  hi-  i  urn-iit 
state  of  the  r(jbot )  and  an  obj(-ctive  of  the  unn-soKa-d  pr<-condil  ion  of  tlx-  push-box  t  hni(  Dimu  ) 
operator  (the  robot  and  box  arc-  at  tlx-  door  of  Room  2)  in  tlx-  satix-  .Move  Robot  probh-m  sjiace 
fB).  In  this  .Move- Robot  probh-m  spac<-  tlx-  got  hru(  l)<M>r)  op<-ralor  is  st-h-cti-c'  x  )  as  the  most  usi-liil 
operator  on  the  path  to  achieve  the  objective  of  getting  tlx-  robot  into  Room  J.  llow(-ver  tlx-  go- 
thru(D(K)r)  operator  also  cannot  apply  and  thus  another  ix-w  probh-m  spaca-  is  i  r<-ati-d  i|))to  la-solve 
its  precondition.  Finally  we  bottom  out  wlx-n  I  lx- go-to(  Dimu-)  op<-ralor  is  si-h-cti-d  iii.md  aituallx 
a[)|)lies  This  application  changes  the  slatt-  of  tlx-  <-xl<'rnal  world  by  bringing  tlx-  robot  to  t  Ix- 
(hxir.  Once  it  is  at  the  d(X)r.  tlx-  termination  cril<-ria  of  tlx-  Move-  Robot  probh-m  sjiai  e  for  t  lx- 
"go-to"  t;usk  is  reached  ((j)and  at  tlx-  saiix-  linx-  the  pnaamdil ions  ol  tlx-  go  thnu  Door)  opertilor  aia- 
resolved  so  it  can  ap[)ly  (U).  Phis  typ»-  of  |)ro< essing  is  rep*-al<'d  until  oix-  ol  tlx-  push  box  i  lirn(  Dimu-  ) 
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Fisnre  19:  SocoikI  Task:  Robol  |)iisliiiip,  a  box 


operator  preconditions  is  met  0.  Note  (hat  we  still  have  not  reached  the  point  that  t  lit'  piish-box- 
thru  operator  can  apply.  So  the  terinination  condition  for  the  .st'cond  Move- Robot  problem  spact' 
has  not  been  reached  and  other  ojrerators  should  b»«  availabi«>  for  st'lection  and  ;ip|)licat ion  in  that 
space.  Thus,  we  see  another  selection  of  tlie  go-lo  opmator (T)ori  the  far  ri,u;ht  tli;il  sends  th('  robot 
to  the  box's  location. 

.Now  that  we  have  two  (a.sk.s  dr'/ined.  what  are  ihe  issni’s  in  combininn  I  hem  so  (hat  wc'  l  an 
juggle  and  move  at  the  same  time? 


4.2  Multiple  tasks  via  interleaving 

We  showed  in  Section  T  that  w<>  couldn't  just  wait  within  an  opmator  lor  a  <|esir(’(l  external  world 
respon.se  because  of  the  uncertainty  of  tlu*  «’xt«'rnal  world  producing  the  desired  |•('sponse.  We 
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then  showed  that  splitting;  an  operator  into  an  aetioii-re<iiiest  <'onii)onent  and  a  (•oinpr<'h(>nd-resnll 
component  removed  the  waiting  time  for  the  desired  result  from  witliin  tin'  operator  and  still 
achieved  the  objective  of  the  o[)erator.  In  this  section,  we  have  a  different  reason  for  lujt  wanting 
to  wait  within  an  operator;  we  want  to  make  progress  on  other  tasks  while  waiting.  Splitting 
the  operator  allows  ns  to  make  progress  on  other  t;isks.  because  it  reiiuwc's  the  operator  for  tin* 
suspended  task  allowing  other  operators  to  be  .s<‘lected.  If  these  opfuators  are  for  other  tasks  then 
the  agent  can  make  progress  on  these  tasks  while  the  first  task  is  suspended. 

The  problem  with  simply  splitting  is  that  the  operators  for  t  lu*  st-cond  task  might  intcu  fere  with 
the  operators  for  the  first  task.  Nonlinear  planning  handles  this  situation  if  the  knowledge  of  which 
tasks  are  being  combined  and  what  all  the  operators  do  is  available.  How('V('r.  sirlitting  r('moves  tin- 
context  of  the  split  operator  and  thus  the  knowledge  that  would  be  t  he  most  useful  to  a  i)lanner.  .\n 
example  interference  scenario  is  the  robot  attempting  to  juggle  and  opi-n  tin-  door  at  the  same  time. 
Since  the  robot  retpiires  the  haiul  that  opens  the  door  to  lx-  empty,  throwing  tin-  bean  bag  to  the 
hand  that  has  been  directed  to  oiren  the  door  interferes  with  opening  the  door,  fikt-wist-  starting 
to  open  the  door  with  a  hand,  might  also  interfert*  with  tin-  catching  of  a  prt'vioiisly  thrown  bean 
bag  to  that  hand.  This  is  not  simply  a  ca.se  of  noidinear  jrlanning.  bc'cause  the  splitting  remove> 
the  easily  checked  context  and  expectations  about  both  tin-  tossing  and  opening  obJ(-ct ives.  .V 
nonlinear  planner  would  use  the  context  to  generate  the  const rtdnts  on  the  action^. 

Controlling  the  interaction,  when  .iplitting.  rerpiires  knowledge  tiboiit  both  the  inti-ntion  to 
catch  the  bean  bag  with  the  hand  and  the  intention  to  o|)en  the  door  with  the  hiiiid.  Onte  w<> 
have  knowledge  of  both  intentions,  then  knowledge  can  be  learned  th;it  constrains  the  solution  so 
that  the  interference  is  avoided.  This  constraining  knowledge  ctui  bt-  gt-nt-ratt-d  bt'fore  i  lu-  scenario 
occurs  by  planning,  or  it  could  be  generatetl  afft-r  the  scenario  occurs  by  rt-playiug  whiii  luippeued. 
Generating  the  con.straining  knowledge  by  planning  i.s  diilicnlt.  bt'caust-  iht-n-  is  tto  )o<  us  to  ilu- 
plan  other  than  time  going  forward.  Let's  go  back  to  our  i-xatiiple  when-  the  beau  btig  is  in  tht- 
air  and  the  agent  wants  to  oiren  the  door  with  the  catching  hand.  I'o  deit-rmiiK-  a  priori  that  the 
catching  hand  should  not  be  ii.sed.  we  have  to  set  up  the  situation  so  iht-  coullict  is  a|)piireni.  1  his 
implies  that  we  have  to  set  u[)  the  conditions  of  the  bean  bag  Ix'iug  dost-  to  tlx-  (ttichiiig  hand. 
But  how  do  we  determine  that  this  is  the  salient  condition  to  set  up.'  Ih-rhaps  the  movt-ment  o| 
the  bean  bag  would  provide  the  necessary  clix'.  Ilowevt'r.  doing  this  by  |)l;iuuiug  does  not  seem  as 
straightforward  as  looking  at  an  error  caused  by  an  interaction. 

The  constraint  for  tossing  to  t  he  hand  o|x'ning  t  he  door  would  be  similar  iii  form  i o  i  he  recover\ 
knowledge  of  Figure  l-o.  It  would  alsi^  recognize  the  intention  of  using  ihecalchiug  hand  lor  opeuim; 
the  door.  The  action  f)f  the  I'onstraint  would  be  to  restrict  the  beau  bag  from  beiiiu,  tossi-d  to  the 
hand  engaged  in  opening  the  door.  Like  th<-  recov(-ry  knowh-dge.  the  const  raiut  kuowh-dgi-  also  has 
to  be  linked  to  the  intention  to  toss  the  bean  bag.  or  it  would  l)e  ov<'r  g<'ueral.  .\s  meutioiied  before, 
these  intentions  coidd  take  many  forms.  I’he  issues  surrounding  th<'ir  exac  t  form  and  persistemc- 
are  the  subject  of  further  research. 


4.3  Multiple  tasks  via  combining  operators 

Combining  operators  nx'ans  taking  tlx-  jxrrtion  of  a  set  of  tasks  reprt-sc'ulc-d  by  a  sc-l  of  opc-rators 
and  deciding  tr)  make  a  single  t  ask  out  of  doing  t  he  o|>erators  togc-t  her.  This  new  t  ask  is  rc-preseui  c-d 
by  a  single  operator  ( we'll  call  it  t  he  ( '-operator)  and  I  he  act  ual  procedure  is  similar  to  the  way  that 
the  (•vclic-tf)ss  o()erator  was  learned  in  l  igiire  2.  Otue  Ix-giin.  if  more- o()eralors  beccmic-  a\ailable. 
these  new  operators  might  join  the  cftmbining  process,  ('ombiniug  operators  Is  import  ant  because 
it  can  cause  previously  independent  tasks  to  be  operatcxl  cm  in  paialh-l.  thus  rc-diiciug  the-  time-  to 
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do  both  tasks.  It  leads  toward  larger  and  larger  operators  that  encoinpass  more  and  more  tasks. 

The  first  issue  with  combining  operators  is  that  the  tasks  they  represent  might  not  l)e  (■omi)letely 
independent.  Non  independent  tasks  interact  with  at  least  some  of  the  actions  in  one  task  affecting 
the  performance  of  the  other  tasks.  This  is  the  same  noidinearity  in  plans  to  reach  conjunctive 
objectives  that  we  saw  in  Section  !. 2.  Here*  we  are  simply  interleaving  in  a  sub-goal.  The  agent  still 
has  the  interacting  task  problem  and  can  either  |)lan  how  to  achieve  all  the  tasks  to  d('t('rmiiie  tin' 
"best  way  to  handle  the  interactions,  or  it  could  a..ssume  interactioi\s  are  uncommon,  and  fi.\  any 
problems  when  they  come  up. 

The  lack  of  simultaneous  completion  of  the  operators  in  the  combination  process  is  tlu*  s(>cond 
issue  with  combining  operators.  .\  range  of  options  is  available  for  tmininating  tin'  cond)inat ion 
process  and  its  associated  ( '-o|)erator.  .\t  oneeml.  termination  occurs  when  all  the  operators  beitiy 
combined  are  completed.  .\t  the  other  end.  we  truininat*'  when  tlu'  first  operator  completes.  W’e 
will  e.xplore  systems  at  both  ends  of  this  range. 

VVe  will  go  over  two  methodologies  for  combining  operators.  The  first  methodology  is  from  [( 'ov- 
rigaru.  1992]  and  stresses  the  planning  aspects  of  cond)ining.  Covrigaru  terminati's  tlu'  ('-operator 
only  when  all  the  operators  in  the  combination  process  have  terminated.  I'he  second  methodology 
is  knowledge-lean.  It  always  starts  the  comlrination  process  when  slack  time  is  ('iicountered  for  an 
operator,  this  initial  operator  is  the  ( '-o()erator.  It  ignores  the  |)lanning  as[)('cts  s(j  it  has  no  need 
of  a  model  of  how  operators  interact.  It  terminates  the  combination  itrocess  when  the  ('-o|)erator 
has  completed  its  original  function,  possibly  leaving  only  i>artially  com|)l(“t('d  ilu'  other  opmators 
being  combined. 


Planning  the  combination 

Covrigaru  combines  operators  in  a  deliberate  way  using  a  strong  model  of  t he  interactions  beiwefui 
actions  and  generating  a  plan  that  o[)timizes  the  kssuing  of  ail  the  actions  from  ihi>  comitining 
operators.  .\  strong  model  lias  interaction  information  for  all  the  |)ossible  interaction  situations. 
He  also  uses  the  interaction  model  when  a  new  operator  becemu's  available  to  decide  if  thi'  opi'raior 
should  be  added  to  the  combination  process.  This  planning  is  the  strength  of  Cov  ligtiru  s  work. 
By  careful  planning  larger  and  larger  PSC.M  operators  are  creati’d  that  encomiiass  the  knowh'dge 
of  dependencies  between  all  the  actions  they  can  issue'.  I'lius  the  actions  tire  not  only  issiu'd  in  an 
implementable  order,  but  in  parallel  when  irossible.  I'ln'  we-akness  of  the  work  is  that  it  reipiiri's 
a  model  of  the  interactions  between  actions  to  do  tin*  planning  and  to  decide  if  a  new  ope-rator 
should  join  the  combination  process. 

Covrigaru  starts  his  process  of  comliining  operators  by  <'xplicitly  creating  a  ru'w  PS(  'M  operator, 
railed  a  “merge"  operator,  wheneve'r  the  correct  conditions  e.xist  for  mi'rging  the  tasks  that  the 
available  operators  represent.  This  merge  operator  exeiutes  the  actions  of  all  I  lu'  to-lx'-mi'rged 
tasks.  Figure  20  gives  an  exam|)le  of  merging  by  showing  three  operators  ( t  hi'  circh's  at  the  loj)  of 
the  triangles)  with  their  associated  actions  (the  sequence  of  hoxi's  under  th<’  o|)erators)  in  the  top 
[)art  of  the  picture.  In  this  task  only  two  of  the  op<>raiors.  push-box-l  hruf  Door  1  .md  toss  (right  ). 
are  initially  available  for  selection.  I'he  operator  toss  (left )  heiomes  seh'ctable  wlii'ii  the'  Ix'an  bag 
is  in  the  left  hand.  Instead  of  jirocessing  the  operators  seipient ially.  a  new  o|)<'rat<)r  is  cri'ati'd  that 
merges  the  actions  of  the  three  operators  as  shown  in  the  bottom  part  ol  Figuri'  20. 

Covrigaru  uses  a  declarative  form  of  the  o|)eralor  actions  and  a  model  of  how  t  In'  ait  ions 
interact  to  control  merging  so  that  the  resulting  operator  issues  the  action  ri'ipii'sts  in  at  li'ast  an 
achievable,  and  [lossibly  optimal,  manni'r.  When  an  action  from  a  sub-operator  bi'comes  available 
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Figjure  '20:  M<>rgiiig;  of  I^SCM  oix'ratois' 

to  Ite  requested  it  is  lieuristically  evaluated  along;  witli  all  the  otln'r  availabh'  actions  to  d('cide  tin' 
next  action  for  the  merge  operator  to  recpiest.  In  our  case,  we  can  arl>itrarily  sen  tin'  heuristic 
evaluation  function  to  prefer  moving  the  robot  over  tossing  a  bean  l)ag.  rinni  ail  the  actions  for 
push-box-thru(  Door)  will  be  preferred  over  toss  (right)  actions.  IJiit  since  push-box-t  hrii(  Door  I 
has  some  slack  time,  its  actions  are  mrt  always  available,  allowing  the  actions  associat('d  with  toss 
( right )  to  be  retpiested  in  svhat  would  have  been  I  he  slack  time  of  t  he  inisli-box-t  hrn(  Door  |  o|)('raior. 

If  a  new  task,  repre'senled  by  an  o|)('rator.  becomes  availal)le  during  tin'  nieiging  proc('ss  t  Inni 
this  new  task  ran  be  added  to  tin'  s('t  ol  tasks  being  nnuged  ami  its  actions  can  also  be  considered 
and  recpiested.  This  happens  in  f  igure  20  when  the  toss  (h'lt)  operatoi-  beconu's  availal)h'  lor 
selection.  The  toss  (right)  operator  has  comfdel<>d  and  the  pnsh-box-l  hrn(door)  op('ralor  has  not 
completed.  How  the  new  task  will  alfc'ct  the  cnrr<'nt  combined  task  is  cin’cked  Ix'lbia'  the  m-w 
task  is  allowed  to  join  the  merging  process,  ('hecking  uses  the  sanu'  tnodel  nsi'd  to  (h'terniine  ihat 
the  original  tasks  shonid  be  merged,  rin*  merge  o|>erator  is  completely  aiipliml  wlnm  all  of  its 
sub-operators  have  comph'tely  applied. 

Figure  20  shows  t  hat  what  is  really  hapixuiing  is  t  he  actual  opt'ratDis  ( t  hat  do  t  he  t  ask  )  are'  Ix-ing 
interh'aved  in  th*'  subgoal.  I’his  interh'aving  is  doin*  nmh'r  the  control  ol  tin'  planning  nn'chanism. 
Chunking  is  converting  this  interh'aving  into  paralh'l  applications,  wln'ii  possible'. 

This  is  a  two  la.sk  mexiilie  ation  of  (>  in  ( 'ovrinarii  [('ovrigani.  IttUj]  wlic  re  I  li<  loss  (li  ll  )  o|irralor  In  i  oiin-s 

available’  for  nn-rging  when  I  In-  loss  (riglil  I  o|i<  rator  lias  I'oiiipicli'il. 


Knowledge-lean  combining 


An  alternative  knowledge-lean  method  is  to  start  combining  ojx'rators  when  slack  time  is  iKjticed. 
Slack  time  causes  an  impasse  to  occur  because  the  definition  of  slack  time  inchuh's  a  lack  of 
knowledge  at  the  PSCM.  This  lack  of  knowledge  can  l)e  handletl  in  the  same  manner  as  all  other 
lacks  of  knowledge  in  the  PSC.M,  namely  by  the  r  reation  <rf  a  pnrblem  sf)ace  to  address  iIk'  lack  of 
knowledge.  The  task  of  this  problem  si)ace  is  to  find  a  PSCM  operation  to  do.  If  operators  from 
a  problem  space  are  eligible  for  selection  then  posing  the  task  that  allows  these  operators  to  Ix' 
selected  is  one  of  the  ways  of  finding  a  PSf'.M  operation  to  do  during  the  slack  time.  Cnfort  unately 
this  method  of  combination  causes  the  function  implemeuti'd  by  the  ('-op('rator  to  Ix'conte  ill- 
defined,  having  side-effects  that  could  not  be  «'xp<'ct(>d  wlu'ii  t  1h‘ operator  is  selectixl  or  terminatf'd. 
It  also  terminates  the  combination  process  when  the  ('-operator  terminates,  causing  the  fanatical 
assumption  to  be  violated  for  the  other  operators  in  the  rombination  process. 

Combining  operators  at  slack  time  causes  the  operator  functions  to  Ix’come  ill-deliix'd  because 
it  overloads  the  meaning  of  the  slack  time  impasse.  When  an  itnpasse  occurs  in  the  I’SC.M  because 
an  operator  cannot  complete,  the  task  that  is  formulated  for  that  impasse  and  tlx'  knowledg(> 
learned  are  supposed  to  resolve  the  lack  of  knowledgi'  that  caused  the  impasse.  In  the  case  of  slack 
time,  a  lack  of  knowledge  e.xists.  but  if  is  a  lack  of  knowledge  as  to  what  to  do  while  waiting  for 
a  response  from  the  external  world,  lii  the  past  we  have  selected  checking  [irogress  on  the  original 
task  as  what  to  do.  But  in  a  multijile  task  model,  we  want  to  consider  working  on  ;uiot Ix'r  task 
as  a  more  reasonable  alternative,  fhe  (‘-operator  in  this  method  is  the  original  operator  that  had 
slack  time.  However,  if  we  select  operators  for  other  tasks  to  run  during  the  slack  time  of  t  Ix' 
(  ’-operator,  knowledge  abotit  those  operators  is  h>arix‘d  in  tlx'  context  of  t  Ix'  C-ojX't  alor.  This 
knowledge  is  independent  of  the  function  the  C-opm-ator  is  impleiix'iit ing  and  can  be  ap|>licabl(’  in 
other  situations.  Thus,  when  the  ('-operator  a|)plies  in  the  futuri'.  it  changes  the  state'  according 
to  its  original  definition,  but  it  also  might  make  other  ciianges  to  the  state  that  are  ixit  really  [lart 
of  its  original  definition.  These  extra  changes  might  evc'ii  caust'  the  original  delinition  to  fail.  This 
would  mean  that  the  ('-operator  and  the  knowledge  learned  from  the  sub-operator  inli'tlert'd  with 
each  other. 

Changing  the  function  that  a  ( '-operatin'  impleiix'iit.s  can  make  pn'viously  h’arix'd  control  know  l¬ 
edge  incorrect.  If  the  control  knowh'dgt*  that  guides  operator  si'h'clion  is  I Ix'  sanx'  lor  all  situations 
then  changing  the  function  implemented  will  work  fiix'.  ix'cause  t  Ix'  nt'w  function  additions  to  tlx' 
('-operator  will  apply  only  when  the  other  op<'rator  is  availalth'  for  seh'clion.  llowi'vt'i.  in  gi'ix'ial 
control  knowledge  responds  to  differi'iit  situations  with  dillert'iil  opi'ralor  seh'itious.  I  hus.  even 
though  the  ('-operator  is  corr<’rtly  seh'ctt'd.  the  auxiliary  changes  it  mak('s  may  not  Ix'  desirable. 
This  is  not  a  itroitiem  for  Covigaru  becaus*'  tlx*  ('-o(x>rator  is  always  a  ix'w  operator  that  is  <|epen- 
dent  upon  a  tie  from  the  smaller  operators,  rims  new  control  information  has  to  be  learned  lor 
the  new  o[)erator. 

Operator  termination  has  a  similar  problem.  In  this  method,  uidiki’  ( 'ovrigiu'u's  combination 
|)roc('ss,  the  op<>rat()rs  being  combined  are  not  treated  e<pially.  In  particular,  tlx-  sub-oper;itor  is 
terminatc'l  when  the  ('-operator  is  completed.  |X)ssibly  belore  the  sub-operator  has  c  (uni>h't('d. 
t'niess  this  non-completion  is  handled,  the  fanatical  operator  afiplicalion  assnm|)lion  will  Ix'  vio 
lated.  If  the  terminations  only  happen  at  slack  time,  ilx-n  we  can  use  tlx-  ie-|)lanning  method  of 
Section  d.  I  to  recover  and  complete  t he  sub-t»peralor.  However,  terminations  can  (xa  ur  .ti  other 
tiiix's  in  Sf)ar  and  the  PSCM  i)laces  ix)  rest rict ions  on  I Ix'se  termiitiil ions.  I  hus.  using  this  iix'thod 
of  cond)ining  makes  implementing  effective  functions  via  operators  impossible  to  guarantee. 

This  nx'tlxxl  of  combining  operators  violates  l«>o  many  of  our  assumptions  as  to  how  PS(’.M 


and  Soar  systems  should  work  without  ofFscttiiig  benefits  to  merit  fiiriher  serious  roiisi()<>i;ii i(j||. 
We  included  it  in  this  paper  because  it  is  an  obvious  metluxl  for  coiubinint'  operators  afiorded  by 
the  Soar  architecture,  and  we  wanted  to  describe  the  problems  with  it. 


5  Discussion 


We  have  introduced  a  number  of  key  ideas  related  to  the  persistence  oi'  objectives,  l  lie  first  is  t  hat 
the  fanatical  completion  assumption  entails  situations  in  which  the  itersisttuice  of  an  o|)erator's 
objective  exceeds  the  desired  persistence  of  the  operator  it.self  on  tlu'Koal  stack.  Wlieti  an  obji'ctivi's 
persistence  exceeds  the  desired  persistence  of  its  operator,  we  s|)lit  tlie  o[)erator.'^  flic  s|)lit  is 
accomplished  by  a  dynamic  redefinition  of  the  opc'rator's  termination  knowledi;('  widcli  may.  of 
necessity,  be  overgeneral.  This  overgenerality  has  two  conseciuences;  it  tnay  preniai  nri'ly  terininate 
the  operator  under  inappropriate  conditions  in  the  future,  and  it  may  cause  a  signilictint  increase  in 
the  number  of  operators  retpiired  to  achieve  an  objective.  Both  these  cons(H|uenc('s  arc'  amc-lioratecl 
somewhat  by  Soar's  delay  of  termination  until  cpiiescence  if  the  world  rc-acts  within  the  l)oiinds 
of  the  decision  cycle,  a  split  operator  effectively  recombines.  Of  course'  the*  procc'ss  of  split  line, 
requires  a  concomitant  process  for  continuing  work  on  the  oltjc'ctive  at  an  appropi ititc'  time  in  tlie 
future.  Specifically,  it  requires  a  potentially  difficult  contc'xt  restoration  procc'ss  and  c  cmsidc'i  tit  ion 
of  the  indexing  involved  in  invoking  the  restoration.  .Vlthongh  wc'  liavc*  givc'ii  some'  idi'as  lor  liow 
to  handle  inde.xing  and  restoration  by  taking  advantage  of  the  contc'.xt  tlitit  is  still  available  during 
the  slack  time  that  leads  to  splitting,  this  is  clearly  an  area  rec|uiring  fiirtlic'r  rc'sc>arc  ii. 

Neither  the  persistence  of  an  objective,  nor  the  context  rc'storation  procc'ss  lias  support  from  t  hc' 
Soar  architecture  at  this  point  in  time.  However,  a  relevant  proposal  li;is  bc'c'ii  made'.  Iiowc'vc'r.  in 
the  form  of  maintaining  several  cotnplete  contexts  on  the*  goal  stac  k  [Rosc'nbloom.  I!)b.{].  Considc'r 
what  multiple  contexts  would  mean  for  the'  cyclic  toss  c'xamph'.  If  wc'  had  multiple  contexts  on 
the  goal  stack,  then  wc  could  maintain  the  context  of  the  cyclic  toss  indc'linitc'ly  wldh'  doitig  ot  lic'r 
[trocessing.  When  the  bean  bag  got  to  the  hand,  the*  cyclic-toss  operator  cccuild  siniftly  continue'. 
Thus,  it  seems  as  if  the  cyclic-toss  operator  woidd  nevc-r  have*  to  split,  and  no  contc'xt  rc'storatioti 
knowledge  wotild  be  required,  rnfortunately.  the*  problem  of  termin.iting  it  cyclic-toss  opc-rator 
that  will  never  complete  remains.  This  is  the  case  of  tossing  t he*  helium  balloon.  I'hc'  tc'rttdnaiicui 
knowledge  in  this  ca.se  is  still  likely  to  be  overgeneral.  Once  it  has  bc'en  crc'atc'd.  c'ithc'r  the'  cyclic  tos> 
operator  will  be  lost  or  some  sort  of  recovery  process  will  have*  to  occur,  rinis  mult  iph'  contc'xts  give' 
us  some  architectural  support  but  don't  solve  the  fundamental  probh'tiis  for  s|)litting  and  rc'covc'iw 

Indeed,  it  may  be  premature  to  consider  architc'cl iiral  supjrort  bc'c  ansc'  the'  issue's  outlinc'd  hc'n> 
have  been  evoked  by  considering  only  a  narrow  range'  of  the  pc'rtinc'ut  phc'uomc'na.  I’lic'  c'xamplc's 
given  here  can  be  thought  of  as  a  mismatch  of  persistc'iicc'  in  I  he  small,  ^  et .  if  wc'  considc'r  t  lic'  broad 
range  of  cjbjectives  held  by  an  agent  that  acts  ovc>r  e.xtc'nch'd  pc'iiods  of  time',  it  m'c'iiis  c  h'ar  that 
most  of  those  objectives  [>ersist  long  Ireyond  the  cluration  of  tlic'  individual  o|)<'iators  intc'udc'd  to 
achieve  them.  Put  another  way.  it  is  not  tin'  norm  that  an  agc'iit  h;is  t  hc'  opportunity  to  carry  out  a 
s('C|n«'nce  of  operators  to  achieve  an  objc'c  tivc'  with  timely  res|)onsc'  from  the'  c'xtc'imal  c'livironmc'iil . 
Instead.  c)ur  clays  are  piinctuatcHl  I.'’  the'  nec'd  to  organize'  cenr  rc'sourcc's  to  compc'iisatc'  for  the' 
delays  between  when  we  form  an  intention  to  achieve  an  objc'clivc'  and  whc'ii  the'  world  prc'sc'nts 
the  crpportunity  to  pursue  the  nc'xl  intentional  step.  VW  might  think  of  this  as  a  mismatch  ol 
persistetice  in  the'  large.  Before  architc'ctiiral  change' can  eeime'.  wc'  must  iirst  undc'istand  how  the' 

"  Wc  ran  also  rrcalc  examples  where  the  piTsisti-m  e  i)f  an  operator's  ol)|e<live  o  -.liorli  r  ill. in  tin  opci  .iior 
persistence,  hill  these  c  a.ses  are'lyjticallv  not  problematic’.  .\n  example  problem  of  this  -ort  ol  pi  rsisli  nc  i  niism.ili  li. 
IS  shown  in  [Akynrek.  lUtt.!]. 


notions  of  splitting  and  rerovnry  can  l)e  spread  across  tlie  current  ar(hit(‘ct nral  ineciianisiiis  for 
persistence  in  a  way  that  makes  some  uniform  sense  for  mismatches  in  the  small  and  in  the  large. 
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